Your agent architecture has a half-life of 6 months — Dan Farrelly, CTO, Inngest

>> Going good. >> Going good. >> All right. There we go. >> All right. There we go. >> All right. There we go. All right. All right. All right. Hello everyone. All right, just checking Hello everyone. All right, just checking Hello everyone. All right, just checking voice. voice. voice. >> All right, good. >> All right, good. >> All right, good. >> All right, well >> All right, well >> All right, well uh my name is Dan uh my name is Dan uh my name is Dan and I'm here to talk to you about how and I'm here to talk to you about how and I'm here to talk to you about how your agent architecture your agent architecture your agent architecture has a half-life of 6 months. has a half-life of 6 months. has a half-life of 6 months. But first But first But first who am I? who am I? who am I? I'm Dan, I'm the CTO and co-founder at I'm Dan, I'm the CTO and co-founder at I'm Dan, I'm the CTO and co-founder at Ingest. Ingest. Ingest. And why should you listen to me? And why should you listen to me? And why should you listen to me? Well, first Well, first Well, first I lead an amazing team over at Ingest. I lead an amazing team over at Ingest. I lead an amazing team over at Ingest. We We build a system that reliably We We build a system that reliably We We build a system that reliably executes anything from executes anything from executes anything from agents to workflows to contact agents to workflows to contact agents to workflows to contact pipelines, whatever you want. pipelines, whatever you want. pipelines, whatever you want. I'm deep building AI infra every I'm deep building AI infra every I'm deep building AI infra every [clears throat] day and on top of that [clears throat] day and on top of that [clears throat] day and on top of that also building agents, not just also building agents, not just also building agents, not just pontificating up here in theory about pontificating up here in theory about pontificating up here in theory about how you should build it. So So speaking of building agents speaking of building agents speaking of building agents if you've been building agents for more if you've been building agents for more if you've been building agents for more than 6 months, you've likely rewritten than 6 months, you've likely rewritten than 6 months, you've likely rewritten something, maybe more than once.

something, maybe more than once. something, maybe more than once. A new model A new model A new model a new framework or framework version a new framework or framework version a new framework or framework version new tool calling standard, a new new tool calling standard, a new new tool calling standard, a new pattern, suddenly your architecture pattern, suddenly your architecture pattern, suddenly your architecture doesn't fit. doesn't fit. doesn't fit. It's not a complaint. It's just the It's not a complaint. It's just the It's not a complaint. It's just the reality. Right, things move faster than reality. Right, things move faster than reality. Right, things move faster than ever. ever. ever. You can see if you've been to any of the You can see if you've been to any of the You can see if you've been to any of the the sessions or keynotes. the sessions or keynotes. the sessions or keynotes. So I want you to think about what you I want you to think about what you shipped 6 months ago shipped 6 months ago shipped 6 months ago and how much of it still runs the and how much of it still runs the and how much of it still runs the way that you originally wrote it. way that you originally wrote it. way that you originally wrote it. Some parts of your code likely survives. Some parts of your code likely survives. Some parts of your code likely survives. But did they survive by accident? Or did But did they survive by accident? Or did But did they survive by accident? Or did you design it that way? Did they survive you design it that way? Did they survive you design it that way? Did they survive by design? How did you actually architect your How did you actually architect your agent? agent? agent? Did you architect it at all? It's okay. Did you architect it at all? It's okay. Did you architect it at all? It's okay. Uh did you use a framework? Did you Uh did you use a framework? Did you Uh did you use a framework? Did you custom roll something? custom roll something? custom roll something? Was the architecture an intentional Was the architecture an intentional Was the architecture an intentional design that you really thought through? design that you really thought through? design that you really thought through? Or just kind of evolved into what you Or just kind of evolved into what you Or just kind of evolved into what you have now?

have now? have now? These are all various ways of where These are all various ways of where These are all various ways of where people are these days. people are these days. people are these days. So, So, So, a lot of folks a lot of folks a lot of folks have talked about harness architecture, have talked about harness architecture, have talked about harness architecture, building harnesses. building harnesses. building harnesses. But mostly, I think a lot of people draw But mostly, I think a lot of people draw But mostly, I think a lot of people draw diagrams and they talk about specific diagrams and they talk about specific diagrams and they talk about specific components. Draw just nice lines. It components. Draw just nice lines. It components. Draw just nice lines. It looks very similar like, you know, kind looks very similar like, you know, kind looks very similar like, you know, kind of simple. of simple. of simple. But I want to talk about the conceptual But I want to talk about the conceptual But I want to talk about the conceptual layers when you're building a system layers when you're building a system layers when you're building a system like this. This is maybe the mental like this. This is maybe the mental like this. This is maybe the mental model. model. model. Not specific components. Not specific components. Not specific components. So, in my opinion, So, in my opinion, So, in my opinion, there are three discrete layers. there are three discrete layers. there are three discrete layers. First, the execution layer. First, the execution layer. First, the execution layer. I think of this as the brain. I think of this as the brain. I think of this as the brain. It's where flow, state, It's where flow, state, It's where flow, state, durability, durability, durability, retries happen. retries happen. retries happen. Then there's the context layer. Then there's the context layer. Then there's the context layer. This is the knowledge. All right, this This is the knowledge. All right, this This is the knowledge. All right, this is models, prompts, tools, memory. is models, prompts, tools, memory. is models, prompts, tools, memory. This is the layer that changes the most.

This is the layer that changes the most. This is the layer that changes the most. And then there's compute. And then there's compute. And then there's compute. There are There are There are um this is the hands, right? This is um this is the hands, right? This is um this is the hands, right? This is sandboxes, runtimes, browsers that sandboxes, runtimes, browsers that sandboxes, runtimes, browsers that you're automating. you're automating. you're automating. I think these layers are important to I think these layers are important to I think these layers are important to consider for the following reason. consider for the following reason. consider for the following reason. It's your half-life. It's your half-life. It's your half-life. And what is half-life? All right, it's a And what is half-life? All right, it's a And what is half-life? All right, it's a scientific term. scientific term. scientific term. And it's a scientific term for the time And it's a scientific term for the time And it's a scientific term for the time that it takes for your for something to that it takes for your for something to that it takes for your for something to decay by half. decay by half. decay by half. And I think that your architecture has a And I think that your architecture has a And I think that your architecture has a half-life, also. half-life, also. half-life, also. So, prompts last weeks, if you're lucky, So, prompts last weeks, if you're lucky, So, prompts last weeks, if you're lucky, maybe maybe a single week. maybe maybe a single week. maybe maybe a single week. Uh the models that you use, months, Uh the models that you use, months, Uh the models that you use, months, again, if you're lucky. again, if you're lucky. again, if you're lucky. Uh Uh Uh but I think that execution can last but I think that execution can last but I think that execution can last years, if you do it right. years, if you do it right. years, if you do it right. So, the problem is that I think that So, the problem is that I think that So, the problem is that I think that most teams couple most teams couple most teams couple everything together. everything together. everything together. And what happens then is that one And what happens then is that one And what happens then is that one layer's half-life kind of leaks and layer's half-life kind of leaks and layer's half-life kind of leaks and drags the other components down. You're drags the other components down. You're drags the other components down. You're building it's technical debt by another building it's technical debt by another building it's technical debt by another name.

name. name. So, my thesis is So, my thesis is So, my thesis is think in layers. think in layers. think in layers. Decouple them. Teams run into a lot of issues with the Teams run into a lot of issues with the approaches that they choose. approaches that they choose. approaches that they choose. You might choose a framework, it might You might choose a framework, it might You might choose a framework, it might feel super magical to get going. feel super magical to get going. feel super magical to get going. You might grab a pre-built harness from You might grab a pre-built harness from You might grab a pre-built harness from one of the frontier labs or wherever. one of the frontier labs or wherever. one of the frontier labs or wherever. And or you might custom roll the entire And or you might custom roll the entire And or you might custom roll the entire system yourself, and it might take system yourself, and it might take system yourself, and it might take months or weeks or whatever. But months or weeks or whatever. But months or weeks or whatever. But I think in a lot of these situations, I think in a lot of these situations, I think in a lot of these situations, the abstractions either are not there at the abstractions either are not there at the abstractions either are not there at all or they're too high-level or the all or they're too high-level or the all or they're too high-level or the layers merge. You know, like layers merge. You know, like layers merge. You know, like orchestration is buried deep inside the orchestration is buried deep inside the orchestration is buried deep inside the chain or inside the framework, and you chain or inside the framework, and you chain or inside the framework, and you don't know what the heck's happening. Uh don't know what the heck's happening. Uh don't know what the heck's happening. Uh state might be kept in a sandbox. Uh state might be kept in a sandbox. Uh state might be kept in a sandbox. Uh retries get tangled with prompts, the retries get tangled with prompts, the retries get tangled with prompts, the prompt logic. prompt logic. prompt logic. And what's hard is you can't re- you And what's hard is you can't re- you And what's hard is you can't re- you can't like swap any of these things out can't like swap any of these things out can't like swap any of these things out without rewriting almost everything.

without rewriting almost everything. without rewriting almost everything. So, I think you need to embrace the So, I think you need to embrace the So, I think you need to embrace the change. Know that things are going to change. Know that things are going to change. Know that things are going to change. Know that things are moving change. Know that things are moving change. Know that things are moving fast. But I think also with that is I fast. But I think also with that is I fast. But I think also with that is I want to focus on a layer that I think is want to focus on a layer that I think is want to focus on a layer that I think is the stable layer here. Where you can the stable layer here. Where you can the stable layer here. Where you can invest and think about and get your invest and think about and get your invest and think about and get your abstractions right, so you don't need to abstractions right, so you don't need to abstractions right, so you don't need to rewrite a large component every 6 rewrite a large component every 6 rewrite a large component every 6 months. So, I'm talking about the months. So, I'm talking about the months. So, I'm talking about the execution layer. execution layer. execution layer. I don't think enough people talk about I don't think enough people talk about I don't think enough people talk about this. this. this. And And And I define it as I define it as I define it as the execution layer as being the system the execution layer as being the system the execution layer as being the system responsible for running your code responsible for running your code responsible for running your code reliably, reliably, reliably, managing how, when, or whether each managing how, when, or whether each managing how, when, or whether each piece of work completes. piece of work completes. piece of work completes. And that's independent of the And that's independent of the And that's independent of the infrastructure that it's on. So, what does this look like in an agent So, what does this look like in an agent architecture? architecture? architecture? So, So, So, the execution layer manages the full the execution layer manages the full the execution layer manages the full life cycle. life cycle. life cycle. You know, plan, call a model, run code, You know, plan, call a model, run code, You know, plan, call a model, run code, invoke a sub-agent, loop, retry, invoke a sub-agent, loop, retry, invoke a sub-agent, loop, retry, coordinate.

coordinate. coordinate. You can swap the model, swap the You can swap the model, swap the You can swap the model, swap the context, swap the sandbox, the execution context, swap the sandbox, the execution context, swap the sandbox, the execution layer should be able to remain the same. layer should be able to remain the same. layer should be able to remain the same. It's just the the fundamentals. And It's just the the fundamentals. And It's just the the fundamentals. And that's how you, I think, decouple and that's how you, I think, decouple and that's how you, I think, decouple and you build the system that it can evolve you build the system that it can evolve you build the system that it can evolve for change. for change. for change. So, let's talk about what the execution So, let's talk about what the execution So, let's talk about what the execution layer layer layer has to do. has to do. has to do. First, First, First, resumability. resumability. resumability. Your execution layer must enable your Your execution layer must enable your Your execution layer must enable your system to pick up after failures without system to pick up after failures without system to pick up after failures without restarting from the beginning. restarting from the beginning. restarting from the beginning. You know, agents are handling You know, agents are handling You know, agents are handling longer-running tasks than ever before. longer-running tasks than ever before. longer-running tasks than ever before. So, they must be able to resume when an So, they must be able to resume when an So, they must be able to resume when an LLM call fails, a tool call fails, LLM call fails, a tool call fails, LLM call fails, a tool call fails, everything flakes out. We all know. everything flakes out. We all know. everything flakes out. We all know. So, if you have a failed reflect step So, if you have a failed reflect step So, if you have a failed reflect step 38, 38, 38, you should be able to retry, wait, you should be able to retry, wait, you should be able to retry, wait, continue onwards, instead of having to continue onwards, instead of having to continue onwards, instead of having to go from the beginning, and you're going go from the beginning, and you're going go from the beginning, and you're going to lose tokens, costs, time, work that to lose tokens, costs, time, work that to lose tokens, costs, time, work that your agent might have completed.

your agent might have completed. your agent might have completed. So, for this this to work, a 3-hour run So, for this this to work, a 3-hour run So, for this this to work, a 3-hour run cannot hold state in memory or in disk. cannot hold state in memory or in disk. cannot hold state in memory or in disk. The state must live outside of the work. The state must live outside of the work. The state must live outside of the work. So, this means that the state must be So, this means that the state must be So, this means that the state must be durable durable durable and external. and external. and external. So, without it, you might cobble So, without it, you might cobble So, without it, you might cobble together maybe some manual together maybe some manual together maybe some manual checkpointing. You might come up with a checkpointing. You might come up with a checkpointing. You might come up with a system that has like a log-based system that has like a log-based system that has like a log-based approach where you're going to be like approach where you're going to be like approach where you're going to be like hydrating the state back if you have to hydrating the state back if you have to hydrating the state back if you have to pick up where you left off. pick up where you left off. pick up where you left off. But, I think a lot of those abstractions But, I think a lot of those abstractions But, I think a lot of those abstractions start leaking into the other layers of start leaking into the other layers of start leaking into the other layers of your harness, right? It gets harder to your harness, right? It gets harder to your harness, right? It gets harder to change. change. change. So, next, a key aspect of the execution So, next, a key aspect of the execution So, next, a key aspect of the execution layer is that it must enable you to layer is that it must enable you to layer is that it must enable you to combine a lot of invocation patterns. combine a lot of invocation patterns. combine a lot of invocation patterns. You're going to need crons. You're going You're going to need crons. You're going You're going to need crons. You're going to need to trigger things with events. to need to trigger things with events. to need to trigger things with events. You're going to need APIs, human in the You're going to need APIs, human in the You're going to need APIs, human in the loop. loop. loop. Subagents or dynamic workflows also must Subagents or dynamic workflows also must Subagents or dynamic workflows also must be possible. be possible. be possible. You're going to need to be able to You're going to need to be able to You're going to need to be able to invoke these things synchronously, invoke these things synchronously, invoke these things synchronously, asynchronously, delaying invocation.

asynchronously, delaying invocation. asynchronously, delaying invocation. And I think the key here is that And I think the key here is that And I think the key here is that flexible flexible flexible execution and orchestration primitives execution and orchestration primitives execution and orchestration primitives will enable you to build the pattern or will enable you to build the pattern or will enable you to build the pattern or the system the architecture that you the system the architecture that you the system the architecture that you actually need. actually need. actually need. So, if not, I think again, your harness So, if not, I think again, your harness So, if not, I think again, your harness logic starts absorbing other concepts logic starts absorbing other concepts logic starts absorbing other concepts like queues, workers, polling, backoff, like queues, workers, polling, backoff, like queues, workers, polling, backoff, scheduling. scheduling. scheduling. And now you end up kind of a mess bad And now you end up kind of a mess bad And now you end up kind of a mess bad abstractions. abstractions. abstractions. So, So, So, lastly, lastly, lastly, execution execution execution needs to provide observability across needs to provide observability across needs to provide observability across your entire session. Not just the LLM your entire session. Not just the LLM your entire session. Not just the LLM calls, the tool calls, but database calls, the tool calls, but database calls, the tool calls, but database errors, permissions issues, um triggers, errors, permissions issues, um triggers, errors, permissions issues, um triggers, performance. performance. performance. So, the full session trace across your So, the full session trace across your So, the full session trace across your entire run is essential. entire run is essential. entire run is essential. So, if you can't see the entirety of a So, if you can't see the entirety of a So, if you can't see the entirety of a trace from the trigger through the whole trace from the trigger through the whole trace from the trigger through the whole stack, stack, stack, it's really hard to debug it, let alone it's really hard to debug it, let alone it's really hard to debug it, let alone improve your agent and keep evolving it.

improve your agent and keep evolving it. improve your agent and keep evolving it. So, So, So, what about sandboxes? Sandboxes are so what about sandboxes? Sandboxes are so what about sandboxes? Sandboxes are so hot. So, hot. So, hot. So, agents need sandboxes. agents need sandboxes. agents need sandboxes. We kind of like settled upon that at We kind of like settled upon that at We kind of like settled upon that at this point in time. They need to execute this point in time. They need to execute this point in time. They need to execute code. code. code. They need to browse. They need to They need to browse. They need to They need to browse. They need to manipulate files. They might work on manipulate files. They might work on manipulate files. They might work on disk. disk. disk. But, a sandbox is ephemeral and But, a sandbox is ephemeral and But, a sandbox is ephemeral and stateless by design. stateless by design. stateless by design. So, using it for durability, snapshots, So, using it for durability, snapshots, So, using it for durability, snapshots, or something in state, I think is an or something in state, I think is an or something in state, I think is an anti-pattern. I think it's a difficult anti-pattern. I think it's a difficult anti-pattern. I think it's a difficult thing where the state gets lost and thing where the state gets lost and thing where the state gets lost and you're kind of trying to peel pull the you're kind of trying to peel pull the you're kind of trying to peel pull the pieces back together. pieces back together. pieces back together. So, I think when you have the execution So, I think when you have the execution So, I think when you have the execution layer separate, the execution layer is layer separate, the execution layer is layer separate, the execution layer is what gives the sandbox its context, its what gives the sandbox its context, its what gives the sandbox its context, its sequence, its durability. sequence, its durability. sequence, its durability. So, I think of the sandbox as the hands So, I think of the sandbox as the hands So, I think of the sandbox as the hands and execution as the brain. and execution as the brain. and execution as the brain. So, So, So, So, what's next? So, what's next? So, what's next? All right, what are the next six months All right, what are the next six months All right, what are the next six months of architectures that you're going to of architectures that you're going to of architectures that you're going to need to handle?

need to handle? need to handle? So, if you're paying attention, you're So, if you're paying attention, you're So, if you're paying attention, you're joining a lot of these sessions, you'll joining a lot of these sessions, you'll joining a lot of these sessions, you'll see what architectures are emerging see what architectures are emerging see what architectures are emerging and what each approach brings new and what each approach brings new and what each approach brings new engineering requirements. engineering requirements. engineering requirements. You know, you have background agents, You know, you have background agents, You know, you have background agents, dynamic workflows, dynamic workflows, dynamic workflows, autonomous loops, autonomous loops, autonomous loops, agent factories, agent factories, agent factories, whatever you want to call it, all these whatever you want to call it, all these whatever you want to call it, all these emerging trends that we're seeing the emerging trends that we're seeing the emerging trends that we're seeing the last couple days. last couple days. last couple days. They're all long-running. They're all long-running. They're all long-running. They're asynchronous. They're asynchronous. They're asynchronous. They're delegated. They're delegated. They're delegated. That means that you need to be able to That means that you need to be able to That means that you need to be able to observe them all observe them all observe them all uh down to the core. All right, they uh down to the core. All right, they uh down to the core. All right, they need to be inspectable by human and by need to be inspectable by human and by need to be inspectable by human and by an agent. an agent. an agent. So, the patterns, as you see, they as So, the patterns, as you see, they as So, the patterns, as you see, they as these things emerge, they must be mixed these things emerge, they must be mixed these things emerge, they must be mixed and combined together. and combined together. and combined together. So, what does this all have in common? So, what does this all have in common? So, what does this all have in common? I think that execution is fundamental to I think that execution is fundamental to I think that execution is fundamental to building any of these systems. building any of these systems. building any of these systems. So, let's just take one example here, So, let's just take one example here, So, let's just take one example here, background agents, right?

background agents, right? background agents, right? They're not request response. There's no They're not request response. There's no They're not request response. There's no person just waiting there for you for person just waiting there for you for person just waiting there for you for the the the chatbot to respond. So, it the the the chatbot to respond. So, it the the the chatbot to respond. So, it could might run for minutes or hours. could might run for minutes or hours. could might run for minutes or hours. It's going to have maybe hundreds of It's going to have maybe hundreds of It's going to have maybe hundreds of calls calls calls to you know, maybe 200 tool calls. to you know, maybe 200 tool calls. to you know, maybe 200 tool calls. You're going to probably guarantee to You're going to probably guarantee to You're going to probably guarantee to have at least one failure in that. So, have at least one failure in that. So, have at least one failure in that. So, you can't even debug your background you can't even debug your background you can't even debug your background agent that's running asynchronously agent that's running asynchronously agent that's running asynchronously without the right infrastructure, without the right infrastructure, without the right infrastructure, without the observability, without without the observability, without without the observability, without everything that's going on in that everything that's going on in that everything that's going on in that process or multiple processes. And another And another example that we have is is loop example that we have is is loop example that we have is is loop architectures. architectures. architectures. Right? I think we're talking about slash Right? I think we're talking about slash Right? I think we're talking about slash loop commands and whatnot and coding loop commands and whatnot and coding loop commands and whatnot and coding agents, but I think where we're taking agents, but I think where we're taking agents, but I think where we're taking this is how are you building this is how are you building this is how are you building actual systems in your products that actual systems in your products that actual systems in your products that employ these these these approaches. employ these these these approaches. employ these these these approaches. So, like what is a loop? So, like what is a loop? So, like what is a loop? Right? A loop is a system that basically Right? A loop is a system that basically Right? A loop is a system that basically just runs continuously or on a schedule just runs continuously or on a schedule just runs continuously or on a schedule and it's assessing the state of the and it's assessing the state of the and it's assessing the state of the system system system against the the goals that you set or against the the goals that you set or against the the goals that you set or the criteria that you set and determines the criteria that you set and determines the criteria that you set and determines what to do next.

what to do next. what to do next. So, you're going to need crons, So, you're going to need crons, So, you're going to need crons, sub-agent delegation, the history needs sub-agent delegation, the history needs sub-agent delegation, the history needs to be inspectable, and of course it to be inspectable, and of course it to be inspectable, and of course it needs to be reliable because you don't needs to be reliable because you don't needs to be reliable because you don't know when things are running or how the know when things are running or how the know when things are running or how the system is continuously continuing to system is continuously continuing to system is continuously continuing to evolve. evolve. evolve. So, the frameworks of 3 months ago So, the frameworks of 3 months ago So, the frameworks of 3 months ago were not designed to handle this. were not designed to handle this. were not designed to handle this. Right? Like you're going to need to Right? Like you're going to need to Right? Like you're going to need to design these systems yourself. design these systems yourself. design these systems yourself. So, I think you're going to need a So, I think you're going to need a So, I think you're going to need a proper execution layer. proper execution layer. proper execution layer. So, let's look at just some examples of So, let's look at just some examples of So, let's look at just some examples of code. It's small on here, but in code. It's small on here, but in code. It's small on here, but in general, let's just look at how we might general, let's just look at how we might general, let's just look at how we might put it together. put it together. put it together. In your loop, you're going to need some In your loop, you're going to need some In your loop, you're going to need some sort of cron. Maybe that cron here is sort of cron. Maybe that cron here is sort of cron. Maybe that cron here is some sort of health check system runs some sort of health check system runs some sort of health check system runs every 30 minutes, pulls down some key every 30 minutes, pulls down some key every 30 minutes, pulls down some key high-level system metrics, and it looks high-level system metrics, and it looks high-level system metrics, and it looks you pass to an LLM. You say, "Is this you pass to an LLM. You say, "Is this you pass to an LLM. You say, "Is this healthy? Is this looking okay? Should we healthy? Is this looking okay? Should we healthy? Is this looking okay? Should we do more? do more? do more? And it is it is healthy or not? What And it is it is healthy or not? What And it is it is healthy or not? What should we do?" should we do?" should we do?" And if it isn't healthy, maybe you just And if it isn't healthy, maybe you just And if it isn't healthy, maybe you just invoke a a triage agent that goes and invoke a a triage agent that goes and invoke a a triage agent that goes and looks at that individual service and looks at that individual service and looks at that individual service and goes and digs more. It's pretty simple goes and digs more. It's pretty simple goes and digs more. It's pretty simple looking, right?

looking, right? looking, right? Now you have this triage agent itself. Now you have this triage agent itself. Now you have this triage agent itself. You've just triggered it. You've just triggered it. You've just triggered it. This should maybe pull some more This should maybe pull some more This should maybe pull some more context, pull some detailed metrics. context, pull some detailed metrics. context, pull some detailed metrics. It's just context. And then you're It's just context. And then you're It's just context. And then you're passing to the LLM to start that passing to the LLM to start that passing to the LLM to start that investigation. Putting in a loop, you're investigation. Putting in a loop, you're investigation. Putting in a loop, you're calling some tools, trying to get to the calling some tools, trying to get to the calling some tools, trying to get to the root cause. This may run for a minute. root cause. This may run for a minute. root cause. This may run for a minute. This may run for multiple minutes. It This may run for multiple minutes. It This may run for multiple minutes. It needs to spin up a sandbox, needs to spin up a sandbox, needs to spin up a sandbox, uh maybe clone code, maybe analyze uh maybe clone code, maybe analyze uh maybe clone code, maybe analyze commits. It's going to be doing a lot of commits. It's going to be doing a lot of commits. It's going to be doing a lot of work here, and you don't know when this work here, and you don't know when this work here, and you don't know when this is going to run. It might run, it might is going to run. It might run, it might is going to run. It might run, it might not. How do you know? How do you observe not. How do you know? How do you observe not. How do you know? How do you observe that system later? that system later? that system later? And then to complete the loop, And then to complete the loop, And then to complete the loop, there you may have a reviewer function, there you may have a reviewer function, there you may have a reviewer function, right? Like, are we Is the system right? Like, are we Is the system right? Like, are we Is the system actually performing as expected? Right? actually performing as expected? Right? actually performing as expected? Right? Like, Like, Like, it's going to run every week and look at it's going to run every week and look at it's going to run every week and look at the history of what just happened the history of what just happened the history of what just happened and evaluate how the triage system is and evaluate how the triage system is and evaluate how the triage system is working. Do we need to adjust the working. Do we need to adjust the working. Do we need to adjust the prompts? Do we need to do better prompts? Do we need to do better prompts? Do we need to do better metrics? Does this have the data that it metrics? Does this have the data that it metrics? Does this have the data that it needs? Is it doing anything at all? Is needs? Is it doing anything at all? Is needs? Is it doing anything at all? Is it overreacting?

it overreacting? it overreacting? So, it needs to be execution aware. So, it needs to be execution aware. So, it needs to be execution aware. And I think it needs to be execution And I think it needs to be execution And I think it needs to be execution aware, orchestration aware, because it aware, orchestration aware, because it aware, orchestration aware, because it needs to be able to pull the logs of the needs to be able to pull the logs of the needs to be able to pull the logs of the system and understand, all right, what system and understand, all right, what system and understand, all right, what was executed? What did this agent choose was executed? What did this agent choose was executed? What did this agent choose to execute? What sub agents did it fan to execute? What sub agents did it fan to execute? What sub agents did it fan out? What workflows did it call upon? out? What workflows did it call upon? out? What workflows did it call upon? And then it needs to be able to analyze And then it needs to be able to analyze And then it needs to be able to analyze that and understand what happened and that and understand what happened and that and understand what happened and come back to you, either make the change come back to you, either make the change come back to you, either make the change itself or tell you this is where we itself or tell you this is where we itself or tell you this is where we performed or underperformed. This is performed or underperformed. This is performed or underperformed. This is what we can do, how we can improve the what we can do, how we can improve the what we can do, how we can improve the system overall. So, this completes this system overall. So, this completes this system overall. So, this completes this improving loop. improving loop. improving loop. It's three functions. It's pretty simple It's three functions. It's pretty simple It's three functions. It's pretty simple by design. It's flexible to iterate on. by design. It's flexible to iterate on. by design. It's flexible to iterate on. I know things get always a lot more I know things get always a lot more I know things get always a lot more complex in production, but I think about complex in production, but I think about complex in production, but I think about when when I think you think about the when when I think you think about the when when I think you think about the layers this way, makes things a little layers this way, makes things a little layers this way, makes things a little bit little bit easier to think about bit little bit easier to think about bit little bit easier to think about building such a complex system that building such a complex system that building such a complex system that feels like it's, you know, something feels like it's, you know, something feels like it's, you know, something that you might not be able to reach that you might not be able to reach that you might not be able to reach right now. right now. right now. So, So, So, in the spirit of the reviewing in the spirit of the reviewing in the spirit of the reviewing functions, functions, functions, what do you think about measuring your what do you think about measuring your what do you think about measuring your agent, how your agent is performing?

agent, how your agent is performing? agent, how your agent is performing? You know, I think what's interesting is You know, I think what's interesting is You know, I think what's interesting is this execution layer sits between what this execution layer sits between what this execution layer sits between what your users' input is and what's your users' input is and what's your users' input is and what's happening throughout the system. happening throughout the system. happening throughout the system. So, user feedback, actions, So, user feedback, actions, So, user feedback, actions, um and the results of the sessions all um and the results of the sessions all um and the results of the sessions all flow through this execution layer. And I flow through this execution layer. And I flow through this execution layer. And I think this makes it uh an ideal place think this makes it uh an ideal place think this makes it uh an ideal place because it becomes the hub for because it becomes the hub for because it becomes the hub for observability. And it allows you to be observability. And it allows you to be observability. And it allows you to be able to score and understand what your able to score and understand what your able to score and understand what your agent is doing. agent is doing. agent is doing. So, to iterate on your application, you So, to iterate on your application, you So, to iterate on your application, you kind of need all this data connected and kind of need all this data connected and kind of need all this data connected and it needs to be aware of how your code is it needs to be aware of how your code is it needs to be aware of how your code is actually executing. Right? So, I think actually executing. Right? So, I think actually executing. Right? So, I think the the linking of this is is extremely the the linking of this is is extremely the the linking of this is is extremely important. important. important. So, So, So, this is what we built Ingest for. We're this is what we built Ingest for. We're this is what we built Ingest for. We're durable execution for AI agents. durable execution for AI agents. durable execution for AI agents. We're the execution layer. We're the execution layer. We're the execution layer. You can plug in any context layer, You can plug in any context layer, You can plug in any context layer, bring a model, framework, tool, any bring a model, framework, tool, any bring a model, framework, tool, any compute layer. Bring whatever sandbox compute layer. Bring whatever sandbox compute layer. Bring whatever sandbox you want. Bring whatever browser you you want. Bring whatever browser you you want. Bring whatever browser you want. You get durable steps, small want. You get durable steps, small want. You get durable steps, small primitives, right? Uh event triggers, primitives, right? Uh event triggers, primitives, right? Uh event triggers, scheduling, scheduling, scheduling, agent-to-agent coordination, full agent-to-agent coordination, full agent-to-agent coordination, full session traces, no infra to manage.

And I think like I just mentioned with And I think like I just mentioned with scores, scores, scores, we also think that that that the middle we also think that that that the middle we also think that that that the middle of the execution layer, when your agent of the execution layer, when your agent of the execution layer, when your agent is running, is running, is running, is a key place to instrument and score is a key place to instrument and score is a key place to instrument and score your agents. your agents. your agents. When you have access to the data that When you have access to the data that When you have access to the data that you need that's all flowing through you need that's all flowing through you need that's all flowing through there or you've just executed something, there or you've just executed something, there or you've just executed something, it's a perfect place to it's a perfect place to it's a perfect place to run something after and defer some run something after and defer some run something after and defer some scoring for with the whole trace scoring for with the whole trace scoring for with the whole trace information, maybe the inputs and the information, maybe the inputs and the information, maybe the inputs and the outputs. So, again, that's just outputs. So, again, that's just outputs. So, again, that's just execution orchestration, delayed task execution orchestration, delayed task execution orchestration, delayed task deferrals deferrals deferrals on on on and since there's also data flowing and since there's also data flowing and since there's also data flowing through, you can also through, you can also through, you can also wait for additional events and attach wait for additional events and attach wait for additional events and attach them to the sessions that you're them to the sessions that you're them to the sessions that you're building and understand did this did building and understand did this did building and understand did this did this actually work, right? If you're if this actually work, right? If you're if this actually work, right? If you're if you're running this triage, um was this you're running this triage, um was this you're running this triage, um was this triage successful? Did it result in an triage successful? Did it result in an triage successful? Did it result in an action by the engineering team? Then action by the engineering team? Then action by the engineering team? Then that means it probably was a positive that means it probably was a positive that means it probably was a positive result. Instead of a thumbs up, thumbs result. Instead of a thumbs up, thumbs result. Instead of a thumbs up, thumbs down, it's like did we open the PR, down, it's like did we open the PR, down, it's like did we open the PR, right? If it's a research agent, was right? If it's a research agent, was right? If it's a research agent, was this research saved? Was it a good this research saved? Was it a good this research saved? Was it a good report? That is these these things that report? That is these these things that report? That is these these things that are events that you should be able to are events that you should be able to are events that you should be able to attach and when you have a system that attach and when you have a system that attach and when you have a system that can connect all these pieces, I think can connect all these pieces, I think can connect all these pieces, I think it's really makes doing a lot of those it's really makes doing a lot of those it's really makes doing a lot of those things things things um like creating outcome-based scores a um like creating outcome-based scores a um like creating outcome-based scores a lot easier.

lot easier. lot easier. So, So, So, to wrap up, to wrap up, to wrap up, build your harness. build your harness. build your harness. You know, understand the layers of the You know, understand the layers of the You know, understand the layers of the architecture. architecture. architecture. We have to embrace fast-paced pace of We have to embrace fast-paced pace of We have to embrace fast-paced pace of change. change. change. And I think if you can get your And I think if you can get your And I think if you can get your execution layer right and think about execution layer right and think about execution layer right and think about the right primitives, the right primitives, the right primitives, everything else can quickly evolve everything else can quickly evolve everything else can quickly evolve around it. And the next 6 months, the around it. And the next 6 months, the around it. And the next 6 months, the next 3 months will all be a lot easier next 3 months will all be a lot easier next 3 months will all be a lot easier for for for especially as everything continues to especially as everything continues to especially as everything continues to change. You all are here. change. You all are here. change. You all are here. So, So, So, thanks for listening. I'm Dan. thanks for listening. I'm Dan. thanks for listening. I'm Dan. Um come and find me at the ng-js booth. Um come and find me at the ng-js booth. Um come and find me at the ng-js booth. We're right on the other side of this We're right on the other side of this We're right on the other side of this wall. We're very bright and orange. So, wall. We're very bright and orange. So, wall. We're very bright and orange. So, thanks everyone. Appreciate it. >> [music] >> [music] >> Mhm.