Every step you take, every call you make: the reliable agent stack — Giselle van Dongen, Restate

Hi everyone. This talk will be about how Hi everyone. This talk will be about how to run agents reliably in production. It to run agents reliably in production. It to run agents reliably in production. It will not be about the eile part, but it will not be about the eile part, but it will not be about the eile part, but it will be about all the other things you will be about all the other things you will be about all the other things you need to get going in order to run agents need to get going in order to run agents need to get going in order to run agents resiliently. So the infrastructure layer resiliently. So the infrastructure layer resiliently. So the infrastructure layer basically. I want to set the scene with basically. I want to set the scene with basically. I want to set the scene with this uh quote of Andre Apathy of last this uh quote of Andre Apathy of last this uh quote of Andre Apathy of last week. It describes that the way we week. It describes that the way we week. It describes that the way we interact with agents and LLMs has been interact with agents and LLMs has been interact with agents and LLMs has been evolving in three waves. The first wave evolving in three waves. The first wave evolving in three waves. The first wave was an LLM being something like a was an LLM being something like a was an LLM being something like a website where we go to we ask it a website where we go to we ask it a website where we go to we ask it a question, it thinks for a few seconds question, it thinks for a few seconds question, it thinks for a few seconds and then gives us a response. The second and then gives us a response. The second and then gives us a response. The second wave was going towards agents. It was an wave was going towards agents. It was an wave was going towards agents. It was an app that we download to our computer. It app that we download to our computer. It app that we download to our computer. It has some tools at its disposal and it has some tools at its disposal and it has some tools at its disposal and it can do some work with our interaction. can do some work with our interaction. can do some work with our interaction. Now the third wave will be going more Now the third wave will be going more Now the third wave will be going more and more towards persistent and and more towards persistent and and more towards persistent and asynchronous entities. So agents being asynchronous entities. So agents being asynchronous entities. So agents being longunning processes in our longunning processes in our longunning processes in our infrastructure with access to tools and infrastructure with access to tools and infrastructure with access to tools and other agents around the organization and other agents around the organization and other agents around the organization and context.

context. context. And so as our use cases are evolving And so as our use cases are evolving And so as our use cases are evolving more and more from single agents to more and more from single agents to more and more from single agents to agentic platforms that connect parts agentic platforms that connect parts agentic platforms that connect parts around uh the organization our around uh the organization our around uh the organization our infrastructure layer should also evolve infrastructure layer should also evolve infrastructure layer should also evolve with that. So when we look at the types with that. So when we look at the types with that. So when we look at the types of tools that are currently out there to of tools that are currently out there to of tools that are currently out there to implement agents, a lot of innovation implement agents, a lot of innovation implement agents, a lot of innovation has been done on sites such as agent has been done on sites such as agent has been done on sites such as agent SDKs and memory. And agent SDKs are SDKs and memory. And agent SDKs are SDKs and memory. And agent SDKs are really cool to implement PC's and get really cool to implement PC's and get really cool to implement PC's and get started quickly, but they don't started quickly, but they don't started quickly, but they don't necessarily help with like connecting necessarily help with like connecting necessarily help with like connecting the distributed bits around an the distributed bits around an the distributed bits around an organization. organization. organization. And if you want to implement more And if you want to implement more And if you want to implement more complex agentic systems, you actually complex agentic systems, you actually complex agentic systems, you actually need all of those things. So that is the need all of those things. So that is the need all of those things. So that is the layer that you see below here where um layer that you see below here where um layer that you see below here where um you have to deploy extra infrastructure. you have to deploy extra infrastructure. you have to deploy extra infrastructure. Uh you need to write things like retry Uh you need to write things like retry Uh you need to write things like retry logic, recovery logic and all of that is logic, recovery logic and all of that is logic, recovery logic and all of that is actually pretty complex to get right but actually pretty complex to get right but actually pretty complex to get right but completely necessary to run longunning completely necessary to run longunning completely necessary to run longunning stateful and distributed processes in stateful and distributed processes in stateful and distributed processes in production.

production. production. So today I want to talk about an So today I want to talk about an So today I want to talk about an open-source framework called restate. open-source framework called restate. open-source framework called restate. And you can see it a bit as a flexible And you can see it a bit as a flexible And you can see it a bit as a flexible durable foundation that lets you build durable foundation that lets you build durable foundation that lets you build any backend. So it's not specific for any backend. So it's not specific for any backend. So it's not specific for agents but a as agents are also just a agents but a as agents are also just a agents but a as agents are also just a type of a backend uh it also works well type of a backend uh it also works well type of a backend uh it also works well for them. The ideas behind restate come for them. The ideas behind restate come for them. The ideas behind restate come from Apache Flink which is a popular from Apache Flink which is a popular from Apache Flink which is a popular distributed stream processing engine and distributed stream processing engine and distributed stream processing engine and also from some of the exarchitects also from some of the exarchitects also from some of the exarchitects behind Meta Score event infra. behind Meta Score event infra. behind Meta Score event infra. So what are the ingredients in restate? So what are the ingredients in restate? So what are the ingredients in restate? Basically four parts. First of all, it Basically four parts. First of all, it Basically four parts. First of all, it makes sure that a single run of an agent makes sure that a single run of an agent makes sure that a single run of an agent is resilient. This is called durable is resilient. This is called durable is resilient. This is called durable execution in the industry. Think about execution in the industry. Think about execution in the industry. Think about things like when an agent runs for a things like when an agent runs for a things like when an agent runs for a week and then crashes. We want to be week and then crashes. We want to be week and then crashes. We want to be able to bring it back and let it able to bring it back and let it able to bring it back and let it continue exactly at the point where it continue exactly at the point where it continue exactly at the point where it failed. We don't want it to start over failed. We don't want it to start over failed. We don't want it to start over from the beginning. from the beginning. from the beginning. Another um area here is running many Another um area here is running many Another um area here is running many concurrent sessions in parallel. Imagine concurrent sessions in parallel. Imagine concurrent sessions in parallel. Imagine running thousands of concurrent agent running thousands of concurrent agent running thousands of concurrent agent sessions at the same time and needing sessions at the same time and needing sessions at the same time and needing needing to make sure that state is needing to make sure that state is needing to make sure that state is always consistent and that different always consistent and that different always consistent and that different agents don't interfere with each other.

agents don't interfere with each other. agents don't interfere with each other. And then going more towards things like And then going more towards things like And then going more towards things like communication between agents, between communication between agents, between communication between agents, between agents and MCP servers and other tools. agents and MCP servers and other tools. agents and MCP servers and other tools. And finally also control, making sure And finally also control, making sure And finally also control, making sure that when an agent for example uh is that when an agent for example uh is that when an agent for example uh is doing um something you don't want it to doing um something you don't want it to doing um something you don't want it to continue or when it's stuck being able continue or when it's stuck being able continue or when it's stuck being able to actually cancel or kill the to actually cancel or kill the to actually cancel or kill the execution. execution. execution. So the way that you can think of it is So the way that you can think of it is So the way that you can think of it is as follows. Restate is basically a as follows. Restate is basically a as follows. Restate is basically a server which runs in front of your agent server which runs in front of your agent server which runs in front of your agent service. So as a separate component it service. So as a separate component it service. So as a separate component it sits there a bit like a like a message sits there a bit like a like a message sits there a bit like a like a message broker or a proxy and when there's a broker or a proxy and when there's a broker or a proxy and when there's a request for your agent restate proxies request for your agent restate proxies request for your agent restate proxies the request to the service and pushes it the request to the service and pushes it the request to the service and pushes it to the service basically and from that to the service basically and from that to the service basically and from that moment there's a connection open moment there's a connection open moment there's a connection open connection between restate and the agent connection between restate and the agent connection between restate and the agent and that connection will basically be a and that connection will basically be a and that connection will basically be a bit like a lifeline for the agent. So as bit like a lifeline for the agent. So as bit like a lifeline for the agent. So as the agent is doing stuff, it sends the agent is doing stuff, it sends the agent is doing stuff, it sends events over to restate and restate will events over to restate and restate will events over to restate and restate will use that journal of events to recover use that journal of events to recover use that journal of events to recover the process after a failure.

the process after a failure. the process after a failure. So from a slightly higher level um So from a slightly higher level um So from a slightly higher level um explanation, you could say that it's explanation, you could say that it's explanation, you could say that it's turning a normal function in your turning a normal function in your turning a normal function in your application into something that is long application into something that is long application into something that is long running, durable, and stateful without running, durable, and stateful without running, durable, and stateful without having to do um a lot of the complex having to do um a lot of the complex having to do um a lot of the complex things you otherwise need to do for things you otherwise need to do for things you otherwise need to do for this. So my talk today will be mainly a this. So my talk today will be mainly a this. So my talk today will be mainly a demo. So I'll be showing you um a demo. So I'll be showing you um a demo. So I'll be showing you um a research agent that is connected to research agent that is connected to research agent that is connected to Slack. Imagine we are like working at Slack. Imagine we are like working at Slack. Imagine we are like working at some company and we want to make an some company and we want to make an some company and we want to make an Slack agent available to all of our Slack agent available to all of our Slack agent available to all of our employees. employees. employees. So if I go here into Slack then can I So if I go here into Slack then can I So if I go here into Slack then can I can here in this channel for example ask can here in this channel for example ask can here in this channel for example ask what is new in AI. what is new in AI. what is new in AI. Now let's have a look at what it's doing Now let's have a look at what it's doing Now let's have a look at what it's doing under the hood. So if I go back here, I under the hood. So if I go back here, I under the hood. So if I go back here, I have here the restate uh UI. This is a have here the restate uh UI. This is a have here the restate uh UI. This is a bit like a cockpit for your agents. So bit like a cockpit for your agents. So bit like a cockpit for your agents. So you can see a registry of all the agents you can see a registry of all the agents you can see a registry of all the agents that are currently registered and you that are currently registered and you that are currently registered and you can also see for example which execution can also see for example which execution can also see for example which execution is currently happening. So here is the is currently happening. So here is the is currently happening. So here is the deep research agent that I spinned up a deep research agent that I spinned up a deep research agent that I spinned up a few seconds ago. We can see what it's few seconds ago. We can see what it's few seconds ago. We can see what it's currently doing. Now it called first an currently doing. Now it called first an currently doing. Now it called first an LLM and then it sent me an answer via LLM and then it sent me an answer via LLM and then it sent me an answer via Slack. This first LLM call was a planner Slack. This first LLM call was a planner Slack. This first LLM call was a planner agent. So what it did is it planned the agent. So what it did is it planned the agent. So what it did is it planned the research and sent me um a list of research and sent me um a list of research and sent me um a list of subtopics that it wants to research.

subtopics that it wants to research. subtopics that it wants to research. Now if I press here approve then this Now if I press here approve then this Now if I press here approve then this will unblock the workflow and will spin will unblock the workflow and will spin will unblock the workflow and will spin up a set of parallel research agents. So up a set of parallel research agents. So up a set of parallel research agents. So this is basically like the classical this is basically like the classical this is basically like the classical deep research workflow, right? You have deep research workflow, right? You have deep research workflow, right? You have a planner then a set of subress research a planner then a set of subress research a planner then a set of subress research agents and then finally someone uh who agents and then finally someone uh who agents and then finally someone uh who writes a report on this like a writer writes a report on this like a writer writes a report on this like a writer agent agent agent and so this journal you see here on the and so this journal you see here on the and so this journal you see here on the left that is basically the events that left that is basically the events that left that is basically the events that get sent from the agent to the restate get sent from the agent to the restate get sent from the agent to the restate server and if this now crashes at some server and if this now crashes at some server and if this now crashes at some point this journal is what will be used point this journal is what will be used point this journal is what will be used to uh recover the execution to the point to uh recover the execution to the point to uh recover the execution to the point where it failed. I don't know if uh where it failed. I don't know if uh where it failed. I don't know if uh there were some errors. I injected a bit there were some errors. I injected a bit there were some errors. I injected a bit of like tool errors in here. Yeah, here of like tool errors in here. Yeah, here of like tool errors in here. Yeah, here you can for example see that um the sub you can for example see that um the sub you can for example see that um the sub agent first did an LLM call then started agent first did an LLM call then started agent first did an LLM call then started doing some web searches and eventually doing some web searches and eventually doing some web searches and eventually uh one of the web searches didn't go uh one of the web searches didn't go uh one of the web searches didn't go through because the API was down and through because the API was down and through because the API was down and then you see here on the right how it then you see here on the right how it then you see here on the right how it got retrieded and eventually completed got retrieded and eventually completed got retrieded and eventually completed successfully. So instead of starting successfully. So instead of starting successfully. So instead of starting over, it uses the journal to recover the over, it uses the journal to recover the over, it uses the journal to recover the progress.

progress. progress. Let's now have a look at what this looks Let's now have a look at what this looks Let's now have a look at what this looks like in code. like in code. like in code. So the basic unit of how you implement So the basic unit of how you implement So the basic unit of how you implement applications in restate is by writing applications in restate is by writing applications in restate is by writing HTTP handlers and those handlers become HTTP handlers and those handlers become HTTP handlers and those handlers become durable by using the restate SDK. So durable by using the restate SDK. So durable by using the restate SDK. So here in this case we have here our deep here in this case we have here our deep here in this case we have here our deep research handler and here as a first research handler and here as a first research handler and here as a first argument we have a restate object argument we have a restate object argument we have a restate object context and the way you can imagine that context and the way you can imagine that context and the way you can imagine that is basically as that uh connection to is basically as that uh connection to is basically as that uh connection to that restate server. whenever I do an that restate server. whenever I do an that restate server. whenever I do an action on this uh restate object, it action on this uh restate object, it action on this uh restate object, it will lead to an event being sent to will lead to an event being sent to will lead to an event being sent to restate. So for example, when I did that restate. So for example, when I did that restate. So for example, when I did that planner LLM call, what actually happened planner LLM call, what actually happened planner LLM call, what actually happened under the hood was it executed here this under the hood was it executed here this under the hood was it executed here this Python function. This is just a simple Python function. This is just a simple Python function. This is just a simple light um light lm like light um light lm like light um light lm like LLM call and the way I made it durable LLM call and the way I made it durable LLM call and the way I made it durable is by wrapping it in restate.run. is by wrapping it in restate.run. is by wrapping it in restate.run. So what happens is by doing these So what happens is by doing these So what happens is by doing these durable steps if this fails somewhere durable steps if this fails somewhere durable steps if this fails somewhere here two hours or two months later it here two hours or two months later it here two hours or two months later it will recover to exactly that point.

will recover to exactly that point. will recover to exactly that point. So that's the idea of durable execution. So that's the idea of durable execution. So that's the idea of durable execution. You're always able to recover a process You're always able to recover a process You're always able to recover a process to where it was. You can also use that to where it was. You can also use that to where it was. You can also use that for other things not necessarily for for other things not necessarily for for other things not necessarily for failure recovery. For example, imagine failure recovery. For example, imagine failure recovery. For example, imagine we want to ask a human to approve we want to ask a human to approve we want to ask a human to approve something and this approval might take something and this approval might take something and this approval might take weeks or a month. this process needs to weeks or a month. this process needs to weeks or a month. this process needs to be able to um to survive restarts and be able to um to survive restarts and be able to um to survive restarts and redeploys uh over those kind of long redeploys uh over those kind of long redeploys uh over those kind of long periods of time and so with durable periods of time and so with durable periods of time and so with durable execution you can actually also uh execution you can actually also uh execution you can actually also uh suspend a function and let bring it back suspend a function and let bring it back suspend a function and let bring it back when it's able to make progress. So in when it's able to make progress. So in when it's able to make progress. So in the case of a human approval what we do the case of a human approval what we do the case of a human approval what we do here is basically we we create a durable here is basically we we create a durable here is basically we we create a durable promise which lives in that journal a promise which lives in that journal a promise which lives in that journal a bit like a suspension point. Then we ask bit like a suspension point. Then we ask bit like a suspension point. Then we ask uh a human to click that button in uh a human to click that button in uh a human to click that button in select as I showed in the beginning and select as I showed in the beginning and select as I showed in the beginning and while we are waiting this process while we are waiting this process while we are waiting this process actually suspends. So if it's running on actually suspends. So if it's running on actually suspends. So if it's running on serverless this is not using uh serverless this is not using uh serverless this is not using uh execution uh time on our functions.

execution uh time on our functions. execution uh time on our functions. Once the response comes in this then Once the response comes in this then Once the response comes in this then gets unblocked and can continue where it gets unblocked and can continue where it gets unblocked and can continue where it left off. So what we see here is a bit left off. So what we see here is a bit left off. So what we see here is a bit like a workflow. It's a set of steps like a workflow. It's a set of steps like a workflow. It's a set of steps that get executed durably. But when we that get executed durably. But when we that get executed durably. But when we think about agents and also the way that think about agents and also the way that think about agents and also the way that Karpathy described it in the tweet, it's Karpathy described it in the tweet, it's Karpathy described it in the tweet, it's more like a persistent stateful entity more like a persistent stateful entity more like a persistent stateful entity that lives for a longer period of time that lives for a longer period of time that lives for a longer period of time that has some memory. Um, so a workflow that has some memory. Um, so a workflow that has some memory. Um, so a workflow is not the nicest way to model this kind is not the nicest way to model this kind is not the nicest way to model this kind of thing. So the way that we can model of thing. So the way that we can model of thing. So the way that we can model this in restate is by using something this in restate is by using something this in restate is by using something called a virtual object. So imagine in called a virtual object. So imagine in called a virtual object. So imagine in the use case that I'm showing this slack the use case that I'm showing this slack the use case that I'm showing this slack research agent. Imagine that I don't research agent. Imagine that I don't research agent. Imagine that I don't want to wait for 10 minutes to give it want to wait for 10 minutes to give it want to wait for 10 minutes to give it some follow-up context or maybe I think some follow-up context or maybe I think some follow-up context or maybe I think about something else that I should have about something else that I should have about something else that I should have told it. Um I want to actually be able told it. Um I want to actually be able told it. Um I want to actually be able to interact with it, not wait till that to interact with it, not wait till that to interact with it, not wait till that research is finished before I can send a research is finished before I can send a research is finished before I can send a follow-up. follow-up. follow-up. And so this is basically what a virtual And so this is basically what a virtual And so this is basically what a virtual object in restate is. It's a bit like a object in restate is. It's a bit like a object in restate is. It's a bit like a stateful actor. It has a unique ID, for stateful actor. It has a unique ID, for stateful actor. It has a unique ID, for example, a session ID. It has uh some example, a session ID. It has uh some example, a session ID. It has uh some key value states that is isolated for key value states that is isolated for key value states that is isolated for that specific session that you can write that specific session that you can write that specific session that you can write to. Uh imagine for example your history to. Uh imagine for example your history to. Uh imagine for example your history of messages and it also has like a set of messages and it also has like a set of messages and it also has like a set of handlers that can execute durable of handlers that can execute durable of handlers that can execute durable functions uh for this session. So here functions uh for this session. So here functions uh for this session. So here the way I implemented this use case that the way I implemented this use case that the way I implemented this use case that I mentioned of interacting with a I mentioned of interacting with a I mentioned of interacting with a running process is as follows. This is a

running process is as follows. This is a running process is as follows. This is a um a bit a session controller. Again, it um a bit a session controller. Again, it um a bit a session controller. Again, it has like this restate object context at has like this restate object context at has like this restate object context at its disposal to do things in a its disposal to do things in a its disposal to do things in a recoverable way. Uh it can write to this recoverable way. Uh it can write to this recoverable way. Uh it can write to this session store. Here it I'm retrieving session store. Here it I'm retrieving session store. Here it I'm retrieving the chat history. the chat history. the chat history. And one thing that's interesting there And one thing that's interesting there And one thing that's interesting there is that in order to run these kind of is that in order to run these kind of is that in order to run these kind of sessions in very high uh paralyzed ways, sessions in very high uh paralyzed ways, sessions in very high uh paralyzed ways, so thousands of sessions at the same so thousands of sessions at the same so thousands of sessions at the same time, we need to make sure that agents time, we need to make sure that agents time, we need to make sure that agents do not interfere with each other. do not interfere with each other. do not interfere with each other. Imagine I'm sending two messages on Imagine I'm sending two messages on Imagine I'm sending two messages on Slack and now two agents are actually Slack and now two agents are actually Slack and now two agents are actually overwriting each other each other's overwriting each other each other's overwriting each other each other's session state. To prevent that, this session state. To prevent that, this session state. To prevent that, this will guarantee that only one execution will guarantee that only one execution will guarantee that only one execution is running at a time. So a second is running at a time. So a second is running at a time. So a second execution will be cued behind the execution will be cued behind the execution will be cued behind the current one. Then let's have a look at how we Then let's have a look at how we implement this like interacting with implement this like interacting with implement this like interacting with another execution. So an execution in another execution. So an execution in another execution. So an execution in reset has a unique identifier and you reset has a unique identifier and you reset has a unique identifier and you can use that identifier to connect to it can use that identifier to connect to it can use that identifier to connect to it from other processes. for example, to from other processes. for example, to from other processes. for example, to retrieve uh the output, but also to retrieve uh the output, but also to retrieve uh the output, but also to cancel it or maybe to signal it being cancel it or maybe to signal it being cancel it or maybe to signal it being injecting a bit of state into an already injecting a bit of state into an already injecting a bit of state into an already running agent loop. And so this is like running agent loop. And so this is like running agent loop. And so this is like a very flexible type of um uh a very flexible type of um uh a very flexible type of um uh capabilities that you can do to capabilities that you can do to capabilities that you can do to implement things like for example implement things like for example implement things like for example signaling an already ongoing agent loop.

signaling an already ongoing agent loop. signaling an already ongoing agent loop. So what we do here is if there is a So what we do here is if there is a So what we do here is if there is a current execution ongoing then we will current execution ongoing then we will current execution ongoing then we will ask an LLM is this like something that ask an LLM is this like something that ask an LLM is this like something that is relevant for the current agent loop. is relevant for the current agent loop. is relevant for the current agent loop. If that is the case inject this via a If that is the case inject this via a If that is the case inject this via a signal if it's not really relevant for signal if it's not really relevant for signal if it's not really relevant for what we're currently doing then cancel what we're currently doing then cancel what we're currently doing then cancel what you're currently doing and start what you're currently doing and start what you're currently doing and start over again with this new information. over again with this new information. over again with this new information. And so this goes a little bit further And so this goes a little bit further And so this goes a little bit further than workflows. it goes a bit more than workflows. it goes a bit more than workflows. it goes a bit more towards like writing persistent stateful towards like writing persistent stateful towards like writing persistent stateful entities that can interact with each entities that can interact with each entities that can interact with each other and have memory at uh their other and have memory at uh their other and have memory at uh their disposal. So let me show you uh how this disposal. So let me show you uh how this disposal. So let me show you uh how this works. So here if I now ask again what works. So here if I now ask again what works. So here if I now ask again what is new in AI and I wait a few seconds is new in AI and I wait a few seconds is new in AI and I wait a few seconds then it should respond again with a then it should respond again with a then it should respond again with a plan. Um and then I can say for example plan. Um and then I can say for example plan. Um and then I can say for example some extra info focus on frontier models some extra info focus on frontier models some extra info focus on frontier models let's say. So once I have the plan I will inject So once I have the plan I will inject that bit of extra state.

Now let's look at the UI of what this is Now let's look at the UI of what this is now doing. So here I have that now doing. So here I have that now doing. So here I have that controller which I just showed. It controller which I just showed. It controller which I just showed. It started calling an LLM to classify uh started calling an LLM to classify uh started calling an LLM to classify uh this new input. this new input. this new input. Once this comes back, it will probably Once this comes back, it will probably Once this comes back, it will probably decide that it should signal it because decide that it should signal it because decide that it should signal it because it's it's still relevant to the research it's it's still relevant to the research it's it's still relevant to the research it's currently doing. So this inject it's currently doing. So this inject it's currently doing. So this inject that new message into the ongoing agent that new message into the ongoing agent that new message into the ongoing agent loop. So let me show you in the deep loop. So let me show you in the deep loop. So let me show you in the deep research agent again. Um so first it research agent again. Um so first it research agent again. Um so first it called an LLM then asked us then we called an LLM then asked us then we called an LLM then asked us then we injected this uh new message of focus on injected this uh new message of focus on injected this uh new message of focus on frontier models and then it uh took that frontier models and then it uh took that frontier models and then it uh took that into account and started over again. into account and started over again. into account and started over again. Here Here Here I can now for example also say something I can now for example also say something I can now for example also say something like uh forget about that research AI policy. research AI policy. And if I send this then the coord And if I send this then the coord And if I send this then the coord coordinator will um decide to cancel the coordinator will um decide to cancel the coordinator will um decide to cancel the ongoing run and start a new one that ongoing run and start a new one that ongoing run and start a new one that will will will research this new topic. And so this research this new topic. And so this research this new topic. And so this cancellation is basically like a signal cancellation is basically like a signal cancellation is basically like a signal that gets um sent down the stack of or that gets um sent down the stack of or that gets um sent down the stack of or the call chain. So if my agent was the call chain. So if my agent was the call chain. So if my agent was already spinning up sub agents first already spinning up sub agents first already spinning up sub agents first those sub aents would be cancelled then those sub aents would be cancelled then those sub aents would be cancelled then uh the controller itself and like that uh the controller itself and like that uh the controller itself and like that it would basically rewind the stack and it would basically rewind the stack and it would basically rewind the stack and give agents also the ability to roll give agents also the ability to roll give agents also the ability to roll back.

back. back. Okay. Okay, so this went a bit more into Okay. Okay, so this went a bit more into Okay. Okay, so this went a bit more into the direction of like stateful the direction of like stateful the direction of like stateful persistent entities that we can interact persistent entities that we can interact persistent entities that we can interact with over longer periods of time. Now with over longer periods of time. Now with over longer periods of time. Now the last part of the demo that I want to the last part of the demo that I want to the last part of the demo that I want to show is um going more towards like being show is um going more towards like being show is um going more towards like being able to write highly customized able to write highly customized able to write highly customized applications. Imagine that we deploy applications. Imagine that we deploy applications. Imagine that we deploy this in production but then a few months this in production but then a few months this in production but then a few months later a new model provider brings out a later a new model provider brings out a later a new model provider brings out a new model for example fabulous and even new model for example fabulous and even new model for example fabulous and even though the model is very good it's also though the model is very good it's also though the model is very good it's also very expensive and we notice that this very expensive and we notice that this very expensive and we notice that this research agent is actually starting to research agent is actually starting to research agent is actually starting to cost a lot. These kind of uh things that cost a lot. These kind of uh things that cost a lot. These kind of uh things that pop up halfway through a project require pop up halfway through a project require pop up halfway through a project require you to then deploy a a lot of new extra you to then deploy a a lot of new extra you to then deploy a a lot of new extra infra or like find a good way to solve infra or like find a good way to solve infra or like find a good way to solve this. This is the kind of things that this. This is the kind of things that this. This is the kind of things that Restate really excels at. It doesn't Restate really excels at. It doesn't Restate really excels at. It doesn't really peg you into a specific way of really peg you into a specific way of really peg you into a specific way of how you should write your application. how you should write your application. how you should write your application. It basically gives you like a durable It basically gives you like a durable It basically gives you like a durable programming model that lets you programming model that lets you programming model that lets you implement an application in the way that implement an application in the way that implement an application in the way that fits for you and also extend it if fits for you and also extend it if fits for you and also extend it if necessary. So first I showed this um LLM necessary. So first I showed this um LLM necessary. So first I showed this um LLM call in the first example as an inline call in the first example as an inline call in the first example as an inline step. It was just a Python function that step. It was just a Python function that step. It was just a Python function that got persisted. But imagine this use case got persisted. But imagine this use case got persisted. But imagine this use case that we want to actually have a bit more that we want to actually have a bit more that we want to actually have a bit more control over those LLM calls. For control over those LLM calls. For control over those LLM calls. For example, what you can do is then pull example, what you can do is then pull example, what you can do is then pull this out into its own handler.

this out into its own handler. this out into its own handler. And this handler can now do things like And this handler can now do things like And this handler can now do things like for example a policy check and then uh for example a policy check and then uh for example a policy check and then uh do the LLM call. And the other agents do the LLM call. And the other agents do the LLM call. And the other agents instead of doing this LLM call inline instead of doing this LLM call inline instead of doing this LLM call inline can now use restates like distributed can now use restates like distributed can now use restates like distributed communication primitives to actually communication primitives to actually communication primitives to actually just call this LLM gateway instead of just call this LLM gateway instead of just call this LLM gateway instead of doing it as an inline step. And this doing it as an inline step. And this doing it as an inline step. And this service fabric that lets you communicate service fabric that lets you communicate service fabric that lets you communicate between agents also gives you some um between agents also gives you some um between agents also gives you some um things like flow control. So we can for things like flow control. So we can for things like flow control. So we can for example say one department is only example say one department is only example say one department is only allowed to run 300 calls to this LLM allowed to run 300 calls to this LLM allowed to run 300 calls to this LLM gateway at the same time. So the reason gateway at the same time. So the reason gateway at the same time. So the reason why I showed this was just to show you a why I showed this was just to show you a why I showed this was just to show you a bit like that. Uh it's basically just a bit like that. Uh it's basically just a bit like that. Uh it's basically just a a resilient foundation. It makes sure a resilient foundation. It makes sure a resilient foundation. It makes sure that your process can uh recover from that your process can uh recover from that your process can uh recover from even a more advanced types of even a more advanced types of even a more advanced types of infrastructure failures, things like infrastructure failures, things like infrastructure failures, things like network partitions and zombie failures. network partitions and zombie failures. network partitions and zombie failures. And um it gives you like tooling to And um it gives you like tooling to And um it gives you like tooling to extend and customize as your use case extend and customize as your use case extend and customize as your use case grows.

grows. grows. Let's go back to the slides to have a Let's go back to the slides to have a Let's go back to the slides to have a little more of an idea of how this thing little more of an idea of how this thing little more of an idea of how this thing is actually implemented on the inside is actually implemented on the inside is actually implemented on the inside because it's actually a pretty because it's actually a pretty because it's actually a pretty interesting um design or architecture. interesting um design or architecture. interesting um design or architecture. So the way it's implemented is basically So the way it's implemented is basically So the way it's implemented is basically by having a a event-driven distributed by having a a event-driven distributed by having a a event-driven distributed log implementation. log implementation. log implementation. So inside the box you basically on one So inside the box you basically on one So inside the box you basically on one side have the clients on the other side side have the clients on the other side side have the clients on the other side the services and inside the box is a log the services and inside the box is a log the services and inside the box is a log which persists all those journal events which persists all those journal events which persists all those journal events and an event loop and that event loop and an event loop and that event loop and an event loop and that event loop basically gets the events from the basically gets the events from the basically gets the events from the service based on what the event is. It service based on what the event is. It service based on what the event is. It either persists some state in the either persists some state in the either persists some state in the embedded state store or it sets a timer embedded state store or it sets a timer embedded state store or it sets a timer or it sends a request to another agent. or it sends a request to another agent. or it sends a request to another agent. And by doing that you basically have a And by doing that you basically have a And by doing that you basically have a durable um foundation for whatever an durable um foundation for whatever an durable um foundation for whatever an application is doing. application is doing. application is doing. The design of this distributed log is The design of this distributed log is The design of this distributed log is heavily inspired by the way that the heavily inspired by the way that the heavily inspired by the way that the core event infra layer at meta works. Uh core event infra layer at meta works. Uh core event infra layer at meta works. Uh it's basically like an iteration on top it's basically like an iteration on top it's basically like an iteration on top of that. Um and some of those architects of that. Um and some of those architects of that. Um and some of those architects are now have designed that for restate are now have designed that for restate are now have designed that for restate as an more generic solution that is as an more generic solution that is as an more generic solution that is available in open source. There are two available in open source. There are two available in open source. There are two important things related to this important things related to this important things related to this architecture that make it interesting.

architecture that make it interesting. architecture that make it interesting. The first one is that it works as a push The first one is that it works as a push The first one is that it works as a push model. So whereas most workflow model. So whereas most workflow model. So whereas most workflow orchestrators actually pull for new orchestrators actually pull for new orchestrators actually pull for new tasks um for pull from the workflow tasks um for pull from the workflow tasks um for pull from the workflow server, restate actually pushes the server, restate actually pushes the server, restate actually pushes the invocations and the benefit you get from invocations and the benefit you get from invocations and the benefit you get from that is that it has a much lower that is that it has a much lower that is that it has a much lower latency. So you can use these kind of latency. So you can use these kind of latency. So you can use these kind of workflow guarantees in functions around workflow guarantees in functions around workflow guarantees in functions around your application and uh have like a your application and uh have like a your application and uh have like a latencies of for example 45 milliseconds latencies of for example 45 milliseconds latencies of for example 45 milliseconds p99 for like a 10-step workflow. p99 for like a 10-step workflow. p99 for like a 10-step workflow. Pushing invocations also works very well Pushing invocations also works very well Pushing invocations also works very well for serverless because they require you for serverless because they require you for serverless because they require you to basically uh send the request and to basically uh send the request and to basically uh send the request and wake up the function. So this design wake up the function. So this design wake up the function. So this design that I show here includes everything you that I show here includes everything you that I show here includes everything you need. It includes uh as well that state need. It includes uh as well that state need. It includes uh as well that state store where we were embedding the state store where we were embedding the state store where we were embedding the state as the UI. It's a single binary so it's as the UI. It's a single binary so it's as the UI. It's a single binary so it's pretty easy to operate as well to run it pretty easy to operate as well to run it pretty easy to operate as well to run it in like a highly available way. You just in like a highly available way. You just in like a highly available way. You just spin it up multiple times and let it spin it up multiple times and let it spin it up multiple times and let it snapshot to object storage.

snapshot to object storage. snapshot to object storage. So restate has six different SDKs. We So restate has six different SDKs. We So restate has six different SDKs. We also have integrations for most of the also have integrations for most of the also have integrations for most of the popular agent frameworks out there. And popular agent frameworks out there. And popular agent frameworks out there. And of course, because it's just like a of course, because it's just like a of course, because it's just like a flexible layer, you can also just use flexible layer, you can also just use flexible layer, you can also just use any LLM SDK and implement custom agents any LLM SDK and implement custom agents any LLM SDK and implement custom agents by just wrapping some steps into uh by just wrapping some steps into uh by just wrapping some steps into uh these SDK constructs. So, it's open these SDK constructs. So, it's open these SDK constructs. So, it's open source. You can self-host it. We also source. You can self-host it. We also source. You can self-host it. We also have a BYOC offering where we deploy have a BYOC offering where we deploy have a BYOC offering where we deploy restate in your cloud account and uh restate in your cloud account and uh restate in your cloud account and uh that gives you the benefit that data that gives you the benefit that data that gives you the benefit that data doesn't leave your cloud account. doesn't leave your cloud account. doesn't leave your cloud account. Otherwise, there's also a managed cloud Otherwise, there's also a managed cloud Otherwise, there's also a managed cloud offering. offering. offering. This was mainly what I wanted to show. This was mainly what I wanted to show. This was mainly what I wanted to show. If you want to explore the code a bit If you want to explore the code a bit If you want to explore the code a bit further, there is here this the GitHub further, there is here this the GitHub further, there is here this the GitHub repo. It's publicly available. If you repo. It's publicly available. If you repo. It's publicly available. If you like the project, then have a look at like the project, then have a look at like the project, then have a look at the restate repo itself. We are hiring the restate repo itself. We are hiring the restate repo itself. We are hiring across the board for all sorts of roles across the board for all sorts of roles across the board for all sorts of roles going from engineering to marketing, going from engineering to marketing, going from engineering to marketing, especially also here in the Bay Area. especially also here in the Bay Area. especially also here in the Bay Area. So, if you're interested in that, uh, So, if you're interested in that, uh, So, if you're interested in that, uh, then definitely check out our careers then definitely check out our careers then definitely check out our careers page. and I will be outside in front of page. and I will be outside in front of page. and I will be outside in front of the conference hall here if you want to the conference hall here if you want to the conference hall here if you want to ask any questions or learn more about ask any questions or learn more about ask any questions or learn more about restate. Thank you very much.