How Forward Deployed Engineering is done at Factory — Eno Reyes

>> This is the forward deployed engineering >> This is the forward deployed engineering track in case you're in the wrong room. track in case you're in the wrong room. track in case you're in the wrong room. Um as you already know, forward deployed Um as you already know, forward deployed Um as you already know, forward deployed engineering is one of the hottest topics engineering is one of the hottest topics engineering is one of the hottest topics in AI. The most important companies on in AI. The most important companies on in AI. The most important companies on the planet are building out massive FTE the planet are building out massive FTE the planet are building out massive FTE teams. So, think OpenAI, Anthropic, teams. So, think OpenAI, Anthropic, teams. So, think OpenAI, Anthropic, Google DeepMind, you get the idea. Google DeepMind, you get the idea. Google DeepMind, you get the idea. Forward deployed engineering was Forward deployed engineering was Forward deployed engineering was pioneered by Palantir many years ago to pioneered by Palantir many years ago to pioneered by Palantir many years ago to embed really strong software engineers embed really strong software engineers embed really strong software engineers directly into their customers' orgs directly into their customers' orgs directly into their customers' orgs uh to implement customize their uh to implement customize their uh to implement customize their platforms around the nuances of the real platforms around the nuances of the real platforms around the nuances of the real world. So, today we brought in some world. So, today we brought in some world. So, today we brought in some amazing speakers from Anthropic, Cursor, amazing speakers from Anthropic, Cursor, amazing speakers from Anthropic, Cursor, Factory, Ramp, Decagon, and many more uh Factory, Ramp, Decagon, and many more uh Factory, Ramp, Decagon, and many more uh to talk about the current state of to talk about the current state of to talk about the current state of forward deployed engineering, how it forward deployed engineering, how it forward deployed engineering, how it works at their companies, and where it's works at their companies, and where it's works at their companies, and where it's going. Our first speaker going. Our first speaker going. Our first speaker Our first speaker is Eno Reyes. He's the Our first speaker is Eno Reyes. He's the Our first speaker is Eno Reyes. He's the co-founder and CTO at Factory, which is co-founder and CTO at Factory, which is co-founder and CTO at Factory, which is building autonomous software engineering building autonomous software engineering building autonomous software engineering agents for enterprise teams. Previously, agents for enterprise teams. Previously, agents for enterprise teams. Previously, he worked in machine learning and he worked in machine learning and he worked in machine learning and software engineering roles at Hugging software engineering roles at Hugging software engineering roles at Hugging Face and Microsoft. Let's give it up for Face and Microsoft. Let's give it up for Face and Microsoft. Let's give it up for Eno. Eno. Eno. >> [applause] >> Yeah, hey everyone. Excited to chat >> Yeah, hey everyone. Excited to chat today. Um and you know, basically I I my today. Um and you know, basically I I my today. Um and you know, basically I I my hope is that at the end of this you guys hope is that at the end of this you guys hope is that at the end of this you guys get a sense of some of the work that get a sense of some of the work that get a sense of some of the work that we're doing on behalf of our customers we're doing on behalf of our customers we're doing on behalf of our customers and with our customers, and the role of and with our customers, and the role of and with our customers, and the role of what we call a deployed engineer should what we call a deployed engineer should what we call a deployed engineer should hopefully be a little bit clearer since hopefully be a little bit clearer since hopefully be a little bit clearer since I think that there are honestly tons of I think that there are honestly tons of I think that there are honestly tons of different models um for different models um for different models um for uh for how this should actually operate uh for how this should actually operate uh for how this should actually operate inside of an org. Um and so inside of an org. Um and so inside of an org. Um and so I think that when when we start I I I do I think that when when we start I I I do I think that when when we start I I I do think that there are some nuances in

think that there are some nuances in think that there are some nuances in sort of like the Palantir era playbook. sort of like the Palantir era playbook. sort of like the Palantir era playbook. Um and I generally the way that forward Um and I generally the way that forward Um and I generally the way that forward deployed goes is I I see that there are deployed goes is I I see that there are deployed goes is I I see that there are lots of different takes on sort of where lots of different takes on sort of where lots of different takes on sort of where forward deployed sits within the org, forward deployed sits within the org, forward deployed sits within the org, how much it interfaces with the actual how much it interfaces with the actual how much it interfaces with the actual product team or the engineering team, product team or the engineering team, product team or the engineering team, how much work is done on behalf of the how much work is done on behalf of the how much work is done on behalf of the customers versus with them, and how much customers versus with them, and how much customers versus with them, and how much work is done on code itself or basically work is done on code itself or basically work is done on code itself or basically like in the software system versus with like in the software system versus with like in the software system versus with the humans and sort of strategizing, the humans and sort of strategizing, the humans and sort of strategizing, right? And so, um generally, I think the right? And so, um generally, I think the right? And so, um generally, I think the there's um there's um there's um in this older model, a lot of the way in this older model, a lot of the way in this older model, a lot of the way that software needed to be built was you that software needed to be built was you that software needed to be built was you needed to go and access that codebase. needed to go and access that codebase. needed to go and access that codebase. You needed to integrate directly in to You needed to integrate directly in to You needed to integrate directly in to data streams or software or products data streams or software or products data streams or software or products that basically you could only access that basically you could only access that basically you could only access behind the curtain of the customer. And behind the curtain of the customer. And behind the curtain of the customer. And so, if you were building something that so, if you were building something that so, if you were building something that was heavily integrated into their was heavily integrated into their was heavily integrated into their environment, yeah, you kind of had the environment, yeah, you kind of had the environment, yeah, you kind of had the need to send and sort of parachute in need to send and sort of parachute in need to send and sort of parachute in individuals into the org. Um individuals into the org. Um individuals into the org. Um but really that has transformed over but really that has transformed over but really that has transformed over time into a role that sort of forks out, time into a role that sort of forks out, time into a role that sort of forks out, and you see a lot of people who are sort and you see a lot of people who are sort and you see a lot of people who are sort of quote-unquote forward deployed of quote-unquote forward deployed of quote-unquote forward deployed engineers or deployed engineers or engineers or deployed engineers or engineers or deployed engineers or applied AI engineers, and it's always applied AI engineers, and it's always applied AI engineers, and it's always it's always a little bit unclear. Are it's always a little bit unclear. Are it's always a little bit unclear. Are they doing maybe professional services they doing maybe professional services they doing maybe professional services work on behalf of their customer? Are work on behalf of their customer? Are work on behalf of their customer? Are they transforming like the product they transforming like the product they transforming like the product around an individual customer? Are they around an individual customer? Are they around an individual customer? Are they just building entirely net new things in just building entirely net new things in just building entirely net new things in the customer's environment, maybe on top the customer's environment, maybe on top the customer's environment, maybe on top of your product? Uh and I think that the

of your product? Uh and I think that the of your product? Uh and I think that the the at least at Factory, we definitely the at least at Factory, we definitely the at least at Factory, we definitely do not want to be doing professional do not want to be doing professional do not want to be doing professional services work on behalf of a customer. services work on behalf of a customer. services work on behalf of a customer. So, if a customer says, "I want to do a So, if a customer says, "I want to do a So, if a customer says, "I want to do a uh modernization of a codebase, uh and uh modernization of a codebase, uh and uh modernization of a codebase, uh and it's you know, I just got quoted from it's you know, I just got quoted from it's you know, I just got quoted from all of the big consulting firms, it's all of the big consulting firms, it's all of the big consulting firms, it's going to cost this much. Could you do going to cost this much. Could you do going to cost this much. Could you do this consulting work for us?" Uh our this consulting work for us?" Uh our this consulting work for us?" Uh our goal is not to go and actually do that goal is not to go and actually do that goal is not to go and actually do that migration on their behalf, even if we migration on their behalf, even if we migration on their behalf, even if we happen to be using our product, right? happen to be using our product, right? happen to be using our product, right? Um and that is because we don't think Um and that is because we don't think Um and that is because we don't think that that actually makes our product that that actually makes our product that that actually makes our product that much better. Uh and ultimately, that much better. Uh and ultimately, that much better. Uh and ultimately, that is a great way way get I'd say a that is a great way way get I'd say a that is a great way way get I'd say a decent amount of revenue, but I don't decent amount of revenue, but I don't decent amount of revenue, but I don't think that that's the way that you can think that that's the way that you can think that that's the way that you can scale a business out enormously, right? scale a business out enormously, right? scale a business out enormously, right? And so, what we've done is we've instead And so, what we've done is we've instead And so, what we've done is we've instead said, we need deployed engineers to be said, we need deployed engineers to be said, we need deployed engineers to be the tip of the spear of the product. And the tip of the spear of the product. And the tip of the spear of the product. And I'm going to do this and then go back. I'm going to do this and then go back. I'm going to do this and then go back. But but really when we say the tip of But but really when we say the tip of But but really when we say the tip of the spear, what we mean is that deployed the spear, what we mean is that deployed the spear, what we mean is that deployed engineers are basically the stream of engineers are basically the stream of engineers are basically the stream of information from our largest and most information from our largest and most information from our largest and most critical customers of the engineering critical customers of the engineering critical customers of the engineering leadership in that org. The leadership in that org. The leadership in that org. The on-the-ground tactical engineers, their on-the-ground tactical engineers, their on-the-ground tactical engineers, their thought process about how software thought process about how software thought process about how software development and AI is actually happening development and AI is actually happening development and AI is actually happening at that org, and then flowing all of at that org, and then flowing all of at that org, and then flowing all of that information back into our product that information back into our product that information back into our product to then rapidly adjust our product in to then rapidly adjust our product in to then rapidly adjust our product in order to then fit into the customer's order to then fit into the customer's order to then fit into the customer's environment better, right? And so, environment better, right? And so, environment better, right? And so, factory really should be when it gets factory really should be when it gets factory really should be when it gets deployed, and I'll talk about what deployed, and I'll talk about what deployed, and I'll talk about what factory is in a second, but we want that factory is in a second, but we want that factory is in a second, but we want that to be effectively self-assembled inside to be effectively self-assembled inside to be effectively self-assembled inside of our customer's environment, right?

of our customer's environment, right? of our customer's environment, right? And then there's a lot of work that goes And then there's a lot of work that goes And then there's a lot of work that goes into understanding that customer's into understanding that customer's into understanding that customer's environment, what the flows that happen, environment, what the flows that happen, environment, what the flows that happen, and ultimately the ROI story. and ultimately the ROI story. and ultimately the ROI story. And what factory really is to our And what factory really is to our And what factory really is to our customers is a set of building blocks customers is a set of building blocks customers is a set of building blocks for building a software for building a software for building a software software factory, right? And so, when we software factory, right? And so, when we software factory, right? And so, when we say software factory, what we mean is say software factory, what we mean is say software factory, what we mean is there's this implicit process that every there's this implicit process that every there's this implicit process that every organization in the world sits on top organization in the world sits on top organization in the world sits on top of, where signals from the outside world of, where signals from the outside world of, where signals from the outside world flow in on one side, and those signals flow in on one side, and those signals flow in on one side, and those signals could be a lot of different things. It could be a lot of different things. It could be a lot of different things. It could be customer conversations, it could be customer conversations, it could be customer conversations, it could be bug reports, it could be could be bug reports, it could be could be bug reports, it could be internal Slack or Teams conversations, internal Slack or Teams conversations, internal Slack or Teams conversations, it could be an executive saying, "We're it could be an executive saying, "We're it could be an executive saying, "We're going to build this thing," right? All going to build this thing," right? All going to build this thing," right? All of these are signals. Some of them have of these are signals. Some of them have of these are signals. Some of them have higher weight than others, and those higher weight than others, and those higher weight than others, and those signals flow in, and humans implicitly signals flow in, and humans implicitly signals flow in, and humans implicitly or explicitly then choose to then or explicitly then choose to then or explicitly then choose to then prioritize, triage, and build plans prioritize, triage, and build plans prioritize, triage, and build plans around those signals. Those plans are around those signals. Those plans are around those signals. Those plans are then converted, typically by software then converted, typically by software then converted, typically by software developers, into changes into some developers, into changes into some developers, into changes into some source of truth, a code base, an source of truth, a code base, an source of truth, a code base, an engineering system. Um and as those engineering system. Um and as those engineering system. Um and as those changes are actually executed on, they changes are actually executed on, they changes are actually executed on, they flow through a validation stage where flow through a validation stage where flow through a validation stage where people maybe review the code, they QA, people maybe review the code, they QA, people maybe review the code, they QA, they assess the security implications, they assess the security implications, they assess the security implications, they uh pass it through automated they uh pass it through automated they uh pass it through automated validation like SAST tools, linters, validation like SAST tools, linters, validation like SAST tools, linters, type checkers, and ultimately when type checkers, and ultimately when type checkers, and ultimately when everything passes, they then ship and everything passes, they then ship and everything passes, they then ship and deploy. And what do you do with deployed deploy. And what do you do with deployed deploy. And what do you do with deployed monitored software? Well, it it monitored software? Well, it it monitored software? Well, it it generates more signals, right? So, this generates more signals, right? So, this generates more signals, right? So, this implicit feedback loop is instrumented

implicit feedback loop is instrumented implicit feedback loop is instrumented very poorly, to be honest, at most very poorly, to be honest, at most very poorly, to be honest, at most organizations. And if you're able to organizations. And if you're able to organizations. And if you're able to take AI and actually transform each of take AI and actually transform each of take AI and actually transform each of these stages of the pipeline and build these stages of the pipeline and build these stages of the pipeline and build an understanding of what the workflow an understanding of what the workflow an understanding of what the workflow looks like at your org from each stage looks like at your org from each stage looks like at your org from each stage to each stage, then you actually can get to each stage, then you actually can get to each stage, then you actually can get to the point where you have a a flow to the point where you have a a flow to the point where you have a a flow through from signal to deploy that has through from signal to deploy that has through from signal to deploy that has no human intervention. Now, importantly, no human intervention. Now, importantly, no human intervention. Now, importantly, that does not mean that humans are not a that does not mean that humans are not a that does not mean that humans are not a part of engineering this system, right? part of engineering this system, right? part of engineering this system, right? But it is that the flow of signal to But it is that the flow of signal to But it is that the flow of signal to deploy is uninterrupted by a human. Um deploy is uninterrupted by a human. Um deploy is uninterrupted by a human. Um and that software factory concept is and that software factory concept is and that software factory concept is obviously not something that can just obviously not something that can just obviously not something that can just snap your fingers and it appears, right? snap your fingers and it appears, right? snap your fingers and it appears, right? Instead, it requires an investment from Instead, it requires an investment from Instead, it requires an investment from the organization. We we like to say this the organization. We we like to say this the organization. We we like to say this is built, not bought, right? But what is built, not bought, right? But what is built, not bought, right? But what the platform that we've built basically the platform that we've built basically the platform that we've built basically provides to people are the canonical one provides to people are the canonical one provides to people are the canonical one model independent agent harness that you model independent agent harness that you model independent agent harness that you need to do this, because if you want to need to do this, because if you want to need to do this, because if you want to build a software factory, if you choose build a software factory, if you choose build a software factory, if you choose to build that software factory in a to build that software factory in a to build that software factory in a vendor locked solution that has like one vendor locked solution that has like one vendor locked solution that has like one model available to it, uh that is going model available to it, uh that is going model available to it, uh that is going to not only be expensive, but two, uh to not only be expensive, but two, uh to not only be expensive, but two, uh there's open questions about model there's open questions about model there's open questions about model independence and like what is the role independence and like what is the role independence and like what is the role of the model provider in dictating what of the model provider in dictating what of the model provider in dictating what you can or cannot build with your you can or cannot build with your you can or cannot build with your software factory, right? Um and if you software factory, right? Um and if you software factory, right? Um and if you also don't own the traces, the data, also don't own the traces, the data, also don't own the traces, the data, everything that flows through your everything that flows through your everything that flows through your software factory, um then you're software factory, um then you're software factory, um then you're probably going to be in trouble as you probably going to be in trouble as you probably going to be in trouble as you start to want to evolve your software start to want to evolve your software start to want to evolve your software factory, right? And so with Droid, the factory, right? And so with Droid, the factory, right? And so with Droid, the hardness that we build, you not only hardness that we build, you not only hardness that we build, you not only have model independence, but you also have model independence, but you also have model independence, but you also have access to every piece of data that

have access to every piece of data that have access to every piece of data that flows in in and out of Droid, alongside flows in in and out of Droid, alongside flows in in and out of Droid, alongside centralized governance and control at centralized governance and control at centralized governance and control at the enterprise layer to be able to the enterprise layer to be able to the enterprise layer to be able to dictate where what information flows dictate where what information flows dictate where what information flows where. Um you can air gap Droid if you where. Um you can air gap Droid if you where. Um you can air gap Droid if you want. Some of our partners um in, you want. Some of our partners um in, you want. Some of our partners um in, you know, the most secure uh environments, know, the most secure uh environments, know, the most secure uh environments, uh think finance, health care, uh gov, uh think finance, health care, uh gov, uh think finance, health care, uh gov, uh they air gap Droid and they run their uh they air gap Droid and they run their uh they air gap Droid and they run their software factories entirely contained. software factories entirely contained. software factories entirely contained. Um Um Um One of our deployed engineers jokes that One of our deployed engineers jokes that One of our deployed engineers jokes that you could run Droid in a submarine if you could run Droid in a submarine if you could run Droid in a submarine if you wanted to. And that's that's you wanted to. And that's that's you wanted to. And that's that's honestly true. And so when we think honestly true. And so when we think honestly true. And so when we think about what the role of this deployed about what the role of this deployed about what the role of this deployed engineer is in that context, which I engineer is in that context, which I engineer is in that context, which I probably should have started with, um probably should have started with, um probably should have started with, um you know, you really need somebody who you know, you really need somebody who you know, you really need somebody who can go in and say, I understand this new can go in and say, I understand this new can go in and say, I understand this new model of building software and I model of building software and I model of building software and I understand the building blocks and the understand the building blocks and the understand the building blocks and the pieces. I can help enable building and pieces. I can help enable building and pieces. I can help enable building and constructing these software factories constructing these software factories constructing these software factories with your team, but I ultimately would with your team, but I ultimately would with your team, but I ultimately would like to one, make it so that our product like to one, make it so that our product like to one, make it so that our product effectively, you know, one click effectively, you know, one click effectively, you know, one click self-assembled into your environment, self-assembled into your environment, self-assembled into your environment, which is needed when you have 45,000 which is needed when you have 45,000 which is needed when you have 45,000 people, maybe hundreds of thousands of people, maybe hundreds of thousands of people, maybe hundreds of thousands of uh of of engineers, maybe you have tens uh of of engineers, maybe you have tens uh of of engineers, maybe you have tens of thousands of code bases. Uh you of thousands of code bases. Uh you of thousands of code bases. Uh you you've got to self-assemble, right? You you've got to self-assemble, right? You you've got to self-assemble, right? You just can't manually install this level just can't manually install this level just can't manually install this level of of complexity. Um and and also on the of of complexity. Um and and also on the of of complexity. Um and and also on the sort of like end loop, sort of like end loop, sort of like end loop, why do all of this, right? I I would why do all of this, right? I I would why do all of this, right? I I would argue that there needs to be an ROI or argue that there needs to be an ROI or argue that there needs to be an ROI or an outcome story that is extremely clear an outcome story that is extremely clear an outcome story that is extremely clear from the beginning so that you can say, from the beginning so that you can say, from the beginning so that you can say, well, we know every code change that

well, we know every code change that well, we know every code change that flows through that gets AI code review, flows through that gets AI code review, flows through that gets AI code review, AI QA, AI security analysis is maybe 87% AI QA, AI security analysis is maybe 87% AI QA, AI security analysis is maybe 87% less likely to hit a bug. And what that less likely to hit a bug. And what that less likely to hit a bug. And what that means is that we can reduce our our bug means is that we can reduce our our bug means is that we can reduce our our bug rate by X, that increases our customer rate by X, that increases our customer rate by X, that increases our customer satisfaction by Y, and that leads to satisfaction by Y, and that leads to satisfaction by Y, and that leads to revenue or growth or new business, revenue or growth or new business, revenue or growth or new business, right? Something needs to flow from this right? Something needs to flow from this right? Something needs to flow from this software factory process to core software factory process to core software factory process to core business goals. And that often is a business goals. And that often is a business goals. And that often is a complex story that requires engineering complex story that requires engineering complex story that requires engineering knowledge, it requires business knowledge, it requires business knowledge, it requires business knowledge. And so, if those are the knowledge. And so, if those are the knowledge. And so, if those are the types of things that you think are types of things that you think are types of things that you think are interesting, that is what deployed interesting, that is what deployed interesting, that is what deployed engineers today are doing for us. engineers today are doing for us. engineers today are doing for us. Um Um Um I've sort of outlined it a little bit I've sort of outlined it a little bit I've sort of outlined it a little bit here, but that teach the model step is here, but that teach the model step is here, but that teach the model step is super important because most super important because most super important because most organizations do not have an autonomy organizations do not have an autonomy organizations do not have an autonomy maturity model. They do not have a road maturity model. They do not have a road maturity model. They do not have a road map, they don't have a conception of map, they don't have a conception of map, they don't have a conception of what it means to truly build an what it means to truly build an what it means to truly build an autonomous software organization, right? autonomous software organization, right? autonomous software organization, right? I think a lot of people ask the I think a lot of people ask the I think a lot of people ask the question, what do the humans do in this question, what do the humans do in this question, what do the humans do in this world, right? For us, we see an world, right? For us, we see an world, right? For us, we see an extremely clear role for humans in extremely clear role for humans in extremely clear role for humans in evolving, refining, and scaling software evolving, refining, and scaling software evolving, refining, and scaling software factories, right? So, you basically the factories, right? So, you basically the factories, right? So, you basically the engineers at a company go from directly engineers at a company go from directly engineers at a company go from directly manipulating software to directly manipulating software to directly manipulating software to directly maintaining and managing a system that maintaining and managing a system that maintaining and managing a system that builds software. And that sort of like builds software. And that sort of like builds software. And that sort of like upgrade in the level of abstraction that upgrade in the level of abstraction that upgrade in the level of abstraction that you operate at is actually very you operate at is actually very you operate at is actually very difficult. And a lot of people difficult. And a lot of people difficult. And a lot of people find it extremely challenging. I would find it extremely challenging. I would find it extremely challenging. I would argue that in fact most people, even argue that in fact most people, even argue that in fact most people, even very thoughtful software engineers, will

very thoughtful software engineers, will very thoughtful software engineers, will have a learning curve in trying to have a learning curve in trying to have a learning curve in trying to shift. The people who I think are are shift. The people who I think are are shift. The people who I think are are well suited for this are DevEx people well suited for this are DevEx people well suited for this are DevEx people who have already been thinking about who have already been thinking about who have already been thinking about enablement of other developers. I think enablement of other developers. I think enablement of other developers. I think product managers who want to become very product managers who want to become very product managers who want to become very technical very quick can become really technical very quick can become really technical very quick can become really great at doing this. And I think that great at doing this. And I think that great at doing this. And I think that generally like people who are used to generally like people who are used to generally like people who are used to to working on teams where to working on teams where to working on teams where >> [clears throat] >> [clears throat] >> [clears throat] >> high quality dev environments were a >> high quality dev environments were a >> high quality dev environments were a priority, you will you will get some of priority, you will you will get some of priority, you will you will get some of the canonical things necessary to enable the canonical things necessary to enable the canonical things necessary to enable these agents to succeed. these agents to succeed. these agents to succeed. Um Um Um I haven't really talked about this last I haven't really talked about this last I haven't really talked about this last one, which is design the workflows. one, which is design the workflows. one, which is design the workflows. And I will get to that in a sec, but I I And I will get to that in a sec, but I I And I will get to that in a sec, but I I think that when I say tip of the spear think that when I say tip of the spear think that when I say tip of the spear of the product, like keep in mind I of the product, like keep in mind I of the product, like keep in mind I really do mean everything that is really do mean everything that is really do mean everything that is happening inside of factory. So, our happening inside of factory. So, our happening inside of factory. So, our product encompasses enterprise controls, product encompasses enterprise controls, product encompasses enterprise controls, the droid harness, the workflows that the droid harness, the workflows that the droid harness, the workflows that run on top of it, the observability run on top of it, the observability run on top of it, the observability tools, the cost controls, the auto model tools, the cost controls, the auto model tools, the cost controls, the auto model routing, the quality of the harness. routing, the quality of the harness. routing, the quality of the harness. Like, all of these are pro- potential Like, all of these are pro- potential Like, all of these are pro- potential opportunities of improvement that you opportunities of improvement that you opportunities of improvement that you will discover when you work very closely will discover when you work very closely will discover when you work very closely in these varied or diverse orgs like how in these varied or diverse orgs like how in these varied or diverse orgs like how to solve. Um, to solve. Um, to solve. Um, so so so making a code base agent ready, right?

making a code base agent ready, right? making a code base agent ready, right? This is a very challenging thing to do. This is a very challenging thing to do. This is a very challenging thing to do. Uh, most organizations have some degree Uh, most organizations have some degree Uh, most organizations have some degree of consistency in how they've chosen to of consistency in how they've chosen to of consistency in how they've chosen to build deterministic validation loops build deterministic validation loops build deterministic validation loops inside of their company, right? So, your inside of their company, right? So, your inside of their company, right? So, your code base runs linters, type checkers, code base runs linters, type checkers, code base runs linters, type checkers, uh, it might run some security scans, uh, it might run some security scans, uh, it might run some security scans, and it's like check mark. Like, it and it's like check mark. Like, it and it's like check mark. Like, it passes or it doesn't. The end end test, passes or it doesn't. The end end test, passes or it doesn't. The end end test, they pass or it doesn't, right? Or they they pass or it doesn't, right? Or they they pass or it doesn't, right? Or they don't. Um, what agent readiness really don't. Um, what agent readiness really don't. Um, what agent readiness really is is it's a measure of how many of is is it's a measure of how many of is is it's a measure of how many of these deterministic validation loops are these deterministic validation loops are these deterministic validation loops are present inside of your code base. Uh, present inside of your code base. Uh, present inside of your code base. Uh, when you have a huge volume of these when you have a huge volume of these when you have a huge volume of these feedback loops, uh, agents are able to feedback loops, uh, agents are able to feedback loops, uh, agents are able to operate for greater periods of time on operate for greater periods of time on operate for greater periods of time on more complex tasks without human more complex tasks without human more complex tasks without human intervention. So, we have like a product intervention. So, we have like a product intervention. So, we have like a product that we call missions, which I'll also that we call missions, which I'll also that we call missions, which I'll also touch on in a sec. But, missions is touch on in a sec. But, missions is touch on in a sec. But, missions is basically an extremely elaborate harness basically an extremely elaborate harness basically an extremely elaborate harness built around the concept of working on built around the concept of working on built around the concept of working on extremely difficult knowledge work extremely difficult knowledge work extremely difficult knowledge work problems that are validatable, right? problems that are validatable, right? problems that are validatable, right? And so, the quality of the output of And so, the quality of the output of And so, the quality of the output of these very long-running harnesses of these very long-running harnesses of these very long-running harnesses of advanced agents is directly proportional advanced agents is directly proportional advanced agents is directly proportional to the degree to which you can validate to the degree to which you can validate to the degree to which you can validate their work. And so, if you introduce the their work. And so, if you introduce the their work. And so, if you introduce the ability to validate at scale, then you ability to validate at scale, then you ability to validate at scale, then you introduce increasing autonomy to the introduce increasing autonomy to the introduce increasing autonomy to the org. So, what we'll look at is we have org. So, what we'll look at is we have org. So, what we'll look at is we have tools that help scan all of these tools that help scan all of these tools that help scan all of these things, but often times, uh, the change things, but often times, uh, the change things, but often times, uh, the change is not so simple. Uh, for I'd say maybe is not so simple. Uh, for I'd say maybe is not so simple. Uh, for I'd say maybe 30 to 40% of the low-hanging fruit, you 30 to 40% of the low-hanging fruit, you 30 to 40% of the low-hanging fruit, you click droid, please fix all of this and click droid, please fix all of this and click droid, please fix all of this and it'll go in and it'll fix it, right? But it'll go in and it'll fix it, right? But it'll go in and it'll fix it, right? But for the other 60% some of them involve for the other 60% some of them involve for the other 60% some of them involve workflow changes. Sometimes humans are workflow changes. Sometimes humans are workflow changes. Sometimes humans are not used to the degree of I would say not used to the degree of I would say not used to the degree of I would say like nitpickiness of these automated

like nitpickiness of these automated like nitpickiness of these automated systems. And so you have to sort of be systems. And so you have to sort of be systems. And so you have to sort of be aware of the aware of the aware of the concerns, the the humans, you have to concerns, the the humans, you have to concerns, the the humans, you have to think about like the way that people are think about like the way that people are think about like the way that people are currently developing systems and say, currently developing systems and say, currently developing systems and say, "How do we introduce some of these more "How do we introduce some of these more "How do we introduce some of these more extreme validation strategies without extreme validation strategies without extreme validation strategies without interrupting the dev flow of the humans interrupting the dev flow of the humans interrupting the dev flow of the humans who are involved in the work?" who are involved in the work?" who are involved in the work?" Um and and I mentioned missions because Um and and I mentioned missions because Um and and I mentioned missions because really I think this is one of the more really I think this is one of the more really I think this is one of the more end game of the agent era at least, end game of the agent era at least, end game of the agent era at least, pre-software factory era. But the more pre-software factory era. But the more pre-software factory era. But the more end game of the agent era style end game of the agent era style end game of the agent era style harnesses where it's simply a long harnesses where it's simply a long harnesses where it's simply a long running harness that has almost no human running harness that has almost no human running harness that has almost no human intervention except for the planning intervention except for the planning intervention except for the planning stage, right? Where you go in and you stage, right? Where you go in and you stage, right? Where you go in and you say, "I would like to have this very say, "I would like to have this very say, "I would like to have this very bounded task. I know that I want to bounded task. I know that I want to bounded task. I know that I want to solve this task and here is what solving solve this task and here is what solving solve this task and here is what solving this task means. I will now basically this task means. I will now basically this task means. I will now basically push a lever of inference until the task push a lever of inference until the task push a lever of inference until the task is complete, right? And so is complete, right? And so is complete, right? And so that is actually unbelievably competent that is actually unbelievably competent that is actually unbelievably competent at solving problems where like is at solving problems where like is at solving problems where like is complete is verifiable. So if you can complete is verifiable. So if you can complete is verifiable. So if you can frame any problem as the set of frame any problem as the set of frame any problem as the set of verification verification verification uh systems that need to validate it, uh systems that need to validate it, uh systems that need to validate it, then you can solve that problem with AI then you can solve that problem with AI then you can solve that problem with AI today. Uh and we've seen this work on today. Uh and we've seen this work on today. Uh and we've seen this work on some pretty insane problem spaces like some pretty insane problem spaces like some pretty insane problem spaces like migrating, you know, 30, 40, 50 million migrating, you know, 30, 40, 50 million migrating, you know, 30, 40, 50 million plus line code bases uh fully plus line code bases uh fully plus line code bases uh fully autonomously, um working on advanced uh autonomously, um working on advanced uh autonomously, um working on advanced uh like deep learning strategies around like deep learning strategies around like deep learning strategies around biomed, health care uh sort of problems, biomed, health care uh sort of problems, biomed, health care uh sort of problems, uh financial institutions that optimize

uh financial institutions that optimize uh financial institutions that optimize equity research where you can actually equity research where you can actually equity research where you can actually build models of different equities and build models of different equities and build models of different equities and sort of analyze and compare and and sort of analyze and compare and and sort of analyze and compare and and build sort of a system that can then build sort of a system that can then build sort of a system that can then back prop and or trade on top of the back prop and or trade on top of the back prop and or trade on top of the those equities. Um like it it's those equities. Um like it it's those equities. Um like it it's mind-blowing to me every day what I what mind-blowing to me every day what I what mind-blowing to me every day what I what I hear people are using with these I hear people are using with these I hear people are using with these tools, but it is not something that you tools, but it is not something that you tools, but it is not something that you can just download, install, and hit can just download, install, and hit can just download, install, and hit play, right? It does require agent play, right? It does require agent play, right? It does require agent readiness. So, if your code base isn't readiness. So, if your code base isn't readiness. So, if your code base isn't agent ready, you won't see any of the agent ready, you won't see any of the agent ready, you won't see any of the success of the most capable AI systems success of the most capable AI systems success of the most capable AI systems in the world today, right? So, this is in the world today, right? So, this is in the world today, right? So, this is why we want people to go in and help our why we want people to go in and help our why we want people to go in and help our customers and say, "Hey, look, you can customers and say, "Hey, look, you can customers and say, "Hey, look, you can solve this actually very difficult solve this actually very difficult solve this actually very difficult problem, but it is going to require a problem, but it is going to require a problem, but it is going to require a different form of investment than you different form of investment than you different form of investment than you were thinking. Less so solving the were thinking. Less so solving the were thinking. Less so solving the problem, more so preparing the problem, more so preparing the problem, more so preparing the environment for verification of the environment for verification of the environment for verification of the problem." problem." problem." And by the way, if you're familiar with And by the way, if you're familiar with And by the way, if you're familiar with how these models are actually trained, how these models are actually trained, how these models are actually trained, like this makes total sense, right? They like this makes total sense, right? They like this makes total sense, right? They they get dense reward when they get post they get dense reward when they get post they get dense reward when they get post trained on all these complex tasks. trained on all these complex tasks. trained on all these complex tasks. Models need dense reward. These Models need dense reward. These Models need dense reward. These verification signals form the basis of verification signals form the basis of verification signals form the basis of that reward that they use to keep them that reward that they use to keep them that reward that they use to keep them on track over a long-term goal-directed on track over a long-term goal-directed on track over a long-term goal-directed problem.

problem. problem. Um Um Um So, So, So, I sort of mentioned this earlier, but I sort of mentioned this earlier, but I sort of mentioned this earlier, but but I think that the the core goal for but I think that the the core goal for but I think that the the core goal for us really is to say, if we can hand over us really is to say, if we can hand over us really is to say, if we can hand over a model to you of how this should this a model to you of how this should this a model to you of how this should this transformation should go, then we should transformation should go, then we should transformation should go, then we should theoretically be able to say, "Let's do theoretically be able to say, "Let's do theoretically be able to say, "Let's do this in a couple of different places, this in a couple of different places, this in a couple of different places, and then let your team actually scale and then let your team actually scale and then let your team actually scale this out across the company." this out across the company." this out across the company." I always use the analogy of if you're I always use the analogy of if you're I always use the analogy of if you're familiar with Walt Disney's Epcot, familiar with Walt Disney's Epcot, familiar with Walt Disney's Epcot, the the theme park. Like basically that the the theme park. Like basically that the the theme park. Like basically that theme park was created originally Disney theme park was created originally Disney theme park was created originally Disney wanted to create like a master planned wanted to create like a master planned wanted to create like a master planned exemplary city. He said, "Look, if I can exemplary city. He said, "Look, if I can exemplary city. He said, "Look, if I can create a city that is the future city, create a city that is the future city, create a city that is the future city, then I can use that as a model to the then I can use that as a model to the then I can use that as a model to the rest of the world cities, and they can rest of the world cities, and they can rest of the world cities, and they can develop entirely new forms of develop entirely new forms of develop entirely new forms of transportation and flourishing." And it transportation and flourishing." And it transportation and flourishing." And it became a theme park. But, what's became a theme park. But, what's became a theme park. But, what's interesting is that in that small interesting is that in that small interesting is that in that small example, a lot of other cities actually example, a lot of other cities actually example, a lot of other cities actually did cite some of the ideas that he was did cite some of the ideas that he was did cite some of the ideas that he was writing down and sharing about what like writing down and sharing about what like writing down and sharing about what like centralized urban transit should look centralized urban transit should look centralized urban transit should look like. And now you have like some more like. And now you have like some more like. And now you have like some more contemporary cities built in the last 50 contemporary cities built in the last 50 contemporary cities built in the last 50 years that basically modeled after that years that basically modeled after that years that basically modeled after that toy example. Um what we want to do is we toy example. Um what we want to do is we toy example. Um what we want to do is we want to make sure that we get some of want to make sure that we get some of want to make sure that we get some of that lesson that if you have a working that lesson that if you have a working that lesson that if you have a working example of a city of the future, of a example of a city of the future, of a example of a city of the future, of a code base of the future, um people are code base of the future, um people are code base of the future, um people are smart. They're clever. Humans will look smart. They're clever. Humans will look smart. They're clever. Humans will look at that and they'll say, "Man, that's at that and they'll say, "Man, that's at that and they'll say, "Man, that's really cool. Let's bring that to my part really cool. Let's bring that to my part really cool. Let's bring that to my part of the code base, right?"

of the code base, right?" of the code base, right?" But if you build too much of an advanced But if you build too much of an advanced But if you build too much of an advanced example, then people will say, "That's a example, then people will say, "That's a example, then people will say, "That's a theme park. That is not at all how the theme park. That is not at all how the theme park. That is not at all how the rest of the world works. I just can't rest of the world works. I just can't rest of the world works. I just can't see how that would apply to the way that see how that would apply to the way that see how that would apply to the way that we currently work today, right?" So it's we currently work today, right?" So it's we currently work today, right?" So it's kind of a delicate balance that you have kind of a delicate balance that you have kind of a delicate balance that you have to walk of building something that to walk of building something that to walk of building something that demonstrates the future is achievable demonstrates the future is achievable demonstrates the future is achievable enough, but ultimately does not scare enough, but ultimately does not scare enough, but ultimately does not scare away uh an org who is thinking, "Man, away uh an org who is thinking, "Man, away uh an org who is thinking, "Man, what is going to be the cost of what is going to be the cost of what is going to be the cost of transforming at this pace, right?" Um transforming at this pace, right?" Um transforming at this pace, right?" Um I always think about that quote, you I always think about that quote, you I always think about that quote, you know, the the future is here, it's just know, the the future is here, it's just know, the the future is here, it's just not evenly distributed. Um there are not evenly distributed. Um there are not evenly distributed. Um there are some code bases, and I I say code bases, some code bases, and I I say code bases, some code bases, and I I say code bases, not even companies, that are truly not even companies, that are truly not even companies, that are truly remarkable. They are effectively uh remarkable. They are effectively uh remarkable. They are effectively uh beginning to run on autopilot. Uh we beginning to run on autopilot. Uh we beginning to run on autopilot. Uh we ourselves have roughly 15 to 20% of what ourselves have roughly 15 to 20% of what ourselves have roughly 15 to 20% of what we call like autonomy, and our autonomy we call like autonomy, and our autonomy we call like autonomy, and our autonomy ratio is like in the upper 80%, which ratio is like in the upper 80%, which ratio is like in the upper 80%, which means the ratio of actions done by means the ratio of actions done by means the ratio of actions done by humans to AI systems before humans to AI systems before humans to AI systems before interruption, right? So our own code interruption, right? So our own code interruption, right? So our own code base is fairly agent-ready, pretty base is fairly agent-ready, pretty base is fairly agent-ready, pretty autonomous, um but uh the code bases of autonomous, um but uh the code bases of autonomous, um but uh the code bases of some of our customers are actually even some of our customers are actually even some of our customers are actually even more autonomous because they operate in more autonomous because they operate in more autonomous because they operate in more constrained uh ways, right? So it's more constrained uh ways, right? So it's more constrained uh ways, right? So it's it's sort of like a uh it is not obvious it's sort of like a uh it is not obvious it's sort of like a uh it is not obvious like who gets 100% autonomy first. I like who gets 100% autonomy first. I like who gets 100% autonomy first. I would argue it's probably very contained would argue it's probably very contained would argue it's probably very contained internal tools. Like we have something internal tools. Like we have something internal tools. Like we have something we call like legal droid, which is our we call like legal droid, which is our we call like legal droid, which is our legal workflow. That is effectively 100% legal workflow. That is effectively 100% legal workflow. That is effectively 100% autonomously maintained, but our like autonomously maintained, but our like autonomously maintained, but our like core harness, uh we do not yet have core harness, uh we do not yet have core harness, uh we do not yet have validators that can validate some of the validators that can validate some of the validators that can validate some of the hard visual problems of a like terminal

hard visual problems of a like terminal hard visual problems of a like terminal based harness. Uh Things like flickering based harness. Uh Things like flickering based harness. Uh Things like flickering are really hard to catch in a verif- in are really hard to catch in a verif- in are really hard to catch in a verif- in a verifiable way. So, we're unable to a verifiable way. So, we're unable to a verifiable way. So, we're unable to close the loop on some of those close the loop on some of those close the loop on some of those challenges. It's an engineering task to challenges. It's an engineering task to challenges. It's an engineering task to build the system that can verify some of build the system that can verify some of build the system that can verify some of those very hard problems. And that might those very hard problems. And that might those very hard problems. And that might give you a picture into sort of like the give you a picture into sort of like the give you a picture into sort of like the weird world of the future where humans weird world of the future where humans weird world of the future where humans are sort of visually our advantages in are sort of visually our advantages in are sort of visually our advantages in being visual, our advantages in having being visual, our advantages in having being visual, our advantages in having context of the outside world provide us context of the outside world provide us context of the outside world provide us a lot of work to do in order to build a lot of work to do in order to build a lot of work to do in order to build these systems. So, who's great at this? these systems. So, who's great at this? these systems. So, who's great at this? I If you are a former founder, for sure I If you are a former founder, for sure I If you are a former founder, for sure you should do this. I think it's like you should do this. I think it's like you should do this. I think it's like the quickest way to basically build out the quickest way to basically build out the quickest way to basically build out I mean each like stage of the SDLC that I mean each like stage of the SDLC that I mean each like stage of the SDLC that Droid has, we think is a billion-dollar Droid has, we think is a billion-dollar Droid has, we think is a billion-dollar business. Like just code review, just business. Like just code review, just business. Like just code review, just incident response, just QA, just incident response, just QA, just incident response, just QA, just testing. Like each of these testing. Like each of these testing. Like each of these you will help define basically the you will help define basically the you will help define basically the nature of these products. nature of these products. nature of these products. If you are someone who is used to tech If you are someone who is used to tech If you are someone who is used to tech communication, right? If you are fluent communication, right? If you are fluent communication, right? If you are fluent in AI, you understand how to speak to in AI, you understand how to speak to in AI, you understand how to speak to every level, you have business acumen, every level, you have business acumen, every level, you have business acumen, you have executive presence, that is you have executive presence, that is you have executive presence, that is another great example of someone who another great example of someone who another great example of someone who should do this. And if you are a systems should do this. And if you are a systems should do this. And if you are a systems thinker, if you love designing systems, thinker, if you love designing systems, thinker, if you love designing systems, if you love closing loops, modeling if you love closing loops, modeling if you love closing loops, modeling data, and understanding how the flow data, and understanding how the flow data, and understanding how the flow through a potentially extremely complex through a potentially extremely complex through a potentially extremely complex org should look, then you are also org should look, then you are also org should look, then you are also someone who would thrive at doing this.

someone who would thrive at doing this. someone who would thrive at doing this. So, if all of this seems interesting, So, if all of this seems interesting, So, if all of this seems interesting, hopefully it does. Please do reach out. hopefully it does. Please do reach out. hopefully it does. Please do reach out. And you can reach out to me directly. And you can reach out to me directly. And you can reach out to me directly. I'm just I'm just I'm just Yeah, I'll say it. It's on the slide. Yeah, I'll say it. It's on the slide. Yeah, I'll say it. It's on the slide. I'm [email protected]. I'm [email protected]. I'm [email protected]. And so, you can just email me directly And so, you can just email me directly And so, you can just email me directly or you can apply on our careers page. or you can apply on our careers page. or you can apply on our careers page. It's called engineer, {comma} deployed. It's called engineer, {comma} deployed. It's called engineer, {comma} deployed. So, that's the role. Hopefully this is So, that's the role. Hopefully this is So, that's the role. Hopefully this is interesting and gives you a taste of interesting and gives you a taste of interesting and gives you a taste of what we're doing at Factory. >> [music] >> [music] >> Woo!