Building Agents Is Trivial Now, Context Is the Next Frontier — Jeff Ng, Unblocked

>> Hi all. >> Hi all. Uh my name is Jeff. I'm a founding Uh my name is Jeff. I'm a founding Uh my name is Jeff. I'm a founding engineer at Unblock, and I'm here to engineer at Unblock, and I'm here to engineer at Unblock, and I'm here to talk to you about how building agents talk to you about how building agents talk to you about how building agents has actually gotten pretty easy. has actually gotten pretty easy. has actually gotten pretty easy. But unfortunately, But unfortunately, But unfortunately, they still get things confidently wrong. they still get things confidently wrong. they still get things confidently wrong. So, 6 months ago, it required a team's So, 6 months ago, it required a team's So, 6 months ago, it required a team's effort and basically a quarter to build effort and basically a quarter to build effort and basically a quarter to build out an agent. out an agent. out an agent. Um an agent is more than just models and Um an agent is more than just models and Um an agent is more than just models and tools. It's the models, the tools, and tools. It's the models, the tools, and tools. It's the models, the tools, and everything required to build out a everything required to build out a everything required to build out a production service. production service. production service. Here are some examples of the different Here are some examples of the different Here are some examples of the different systems that were necessary in order to systems that were necessary in order to systems that were necessary in order to build something out. build something out. build something out. Each one of these was basically its own Each one of these was basically its own Each one of these was basically its own company or at least a company function. company or at least a company function. company or at least a company function. Not going to go through each one of Not going to go through each one of Not going to go through each one of these, but you know, a few that stood these, but you know, a few that stood these, but you know, a few that stood out to me. out to me. out to me. First one, checkpoint and state First one, checkpoint and state First one, checkpoint and state persistence. persistence. persistence. Agent runs, they're typically long-lived Agent runs, they're typically long-lived Agent runs, they're typically long-lived and stateful. and stateful. and stateful. Um Um Um unfortunately, unfortunately, unfortunately, uh infrastructure itself though, those uh infrastructure itself though, those uh infrastructure itself though, those that's ephemeral. that's ephemeral. that's ephemeral. Crashing without durability can actually Crashing without durability can actually Crashing without durability can actually lead to a lot of state loss. And that lead to a lot of state loss. And that lead to a lot of state loss. And that state kind of includes things like state kind of includes things like state kind of includes things like message history, message history, message history, tool calls, as well as, you know, where tool calls, as well as, you know, where tool calls, as well as, you know, where you are in the loop.

you are in the loop. you are in the loop. Without these things, you can't resume Without these things, you can't resume Without these things, you can't resume the session. the session. the session. Uh one option is, you know, maybe you Uh one option is, you know, maybe you Uh one option is, you know, maybe you want to restart the session. want to restart the session. want to restart the session. Unfortunately, that's actually quite Unfortunately, that's actually quite Unfortunately, that's actually quite expensive as well. expensive as well. expensive as well. Uh you lose out on all the tokens that Uh you lose out on all the tokens that Uh you lose out on all the tokens that you'd originally used, you'd originally used, you'd originally used, um um um as well as, you know, as well as, you know, as well as, you know, latency. Uh from a user experience latency. Uh from a user experience latency. Uh from a user experience standpoint, you've already triggered standpoint, you've already triggered standpoint, you've already triggered that session. Now you have to wait for that session. Now you have to wait for that session. Now you have to wait for the whole thing to go again. the whole thing to go again. the whole thing to go again. And lastly, side effects. Your agent And lastly, side effects. Your agent And lastly, side effects. Your agent might have performed some side effects, might have performed some side effects, might have performed some side effects, and now there's a chance of those and now there's a chance of those and now there's a chance of those doubling up. doubling up. doubling up. So, So, So, next thing, sandbox infrastructure, next thing, sandbox infrastructure, next thing, sandbox infrastructure, right? So, as we all know, we're running right? So, as we all know, we're running right? So, as we all know, we're running more and more agent-generated code as more and more agent-generated code as more and more agent-generated code as well as third-party code. This gets all well as third-party code. This gets all well as third-party code. This gets all run on your infrastructure, and run on your infrastructure, and run on your infrastructure, and due to that, there are some due to that, there are some due to that, there are some complexities. Uh because of that, complexities. Uh because of that, complexities. Uh because of that, we want to introduce isolated sandboxes, we want to introduce isolated sandboxes, we want to introduce isolated sandboxes, which help prevent which help prevent which help prevent uh unnecessary reads of environment uh unnecessary reads of environment uh unnecessary reads of environment secrets, secrets, secrets, unnecessary network access, unnecessary network access, unnecessary network access, you know, just in general, we don't want you know, just in general, we don't want you know, just in general, we don't want to take down the shared host.

And then, observability. And then, observability. How do we answer the question, "Where How do we answer the question, "Where How do we answer the question, "Where did this fail?" did this fail?" did this fail?" Typically, this includes tracking Typically, this includes tracking Typically, this includes tracking logs and traces from across half a dozen logs and traces from across half a dozen logs and traces from across half a dozen systems. systems. systems. Everything I've mentioned here, none of Everything I've mentioned here, none of Everything I've mentioned here, none of this actually improves an agent's this actually improves an agent's this actually improves an agent's capabilities. capabilities. capabilities. They're all taxes one has to pay in They're all taxes one has to pay in They're all taxes one has to pay in order to get an agent out there to play order to get an agent out there to play order to get an agent out there to play the game. Thankfully, things have changed quite a Thankfully, things have changed quite a bit. bit. bit. Um the whole ecosystem has matured quite Um the whole ecosystem has matured quite Um the whole ecosystem has matured quite a bit, and cloud infrastructure players a bit, and cloud infrastructure players a bit, and cloud infrastructure players such as Cloudflare, uh Vercel, AWS, such as Cloudflare, uh Vercel, AWS, such as Cloudflare, uh Vercel, AWS, they've gone and taken some of that they've gone and taken some of that they've gone and taken some of that complexity away and built primitives complexity away and built primitives complexity away and built primitives that these frameworks, Flu, Vercel E that these frameworks, Flu, Vercel E that these frameworks, Flu, Vercel E Maestra, Maestra, Maestra, with these together, you know, they've with these together, you know, they've with these together, you know, they've taken a lot of complexity away, and you taken a lot of complexity away, and you taken a lot of complexity away, and you can focus more on building the actual can focus more on building the actual can focus more on building the actual agent itself. The core logic that agent itself. The core logic that agent itself. The core logic that actually helps you actually helps you actually helps you and your team and your customers.

So, So, here's an example of one. here's an example of one. here's an example of one. Uh I played around with Flu and Uh I played around with Flu and Uh I played around with Flu and Cloudflare, Cloudflare, Cloudflare, and as you can see on the left-hand and as you can see on the left-hand and as you can see on the left-hand side, side, side, you know, we basically handle everything you know, we basically handle everything you know, we basically handle everything as mentioned before. as mentioned before. as mentioned before. So, the primitives plus the framework So, the primitives plus the framework So, the primitives plus the framework lead to a situation where lead to a situation where lead to a situation where it's actually not that much code to it's actually not that much code to it's actually not that much code to define an agent. define an agent. define an agent. Uh one of the things I was shocked at Uh one of the things I was shocked at Uh one of the things I was shocked at when I first took a look at the when I first took a look at the when I first took a look at the documentation. documentation. documentation. To get in the details, all you really To get in the details, all you really To get in the details, all you really have to do when defining agent is A, have to do when defining agent is A, have to do when defining agent is A, deciding which model you want to use. deciding which model you want to use. deciding which model you want to use. B, the instructions or, you know, the B, the instructions or, you know, the B, the instructions or, you know, the system prompt. system prompt. system prompt. C, the tools that you want to ask the C, the tools that you want to ask the C, the tools that you want to ask the agent to have access to. agent to have access to. agent to have access to. Skills, the things I can do. Skills, the things I can do. Skills, the things I can do. As well as the sandbox location, where As well as the sandbox location, where As well as the sandbox location, where things are being run. things are being run. things are being run. So, So, So, uh uh uh to give you an example of this, to give you an example of this, to give you an example of this, I've actually gone and built out a issue I've actually gone and built out a issue I've actually gone and built out a issue enrichment system specifically for enrichment system specifically for enrichment system specifically for Linear. Linear. Linear. So, what this does is, given a Linear So, what this does is, given a Linear So, what this does is, given a Linear ticket and access to your code ticket and access to your code ticket and access to your code repository, repository, repository, it'll go out, you know, fetch the Linear it'll go out, you know, fetch the Linear it'll go out, you know, fetch the Linear ticket, determine whether or not it's a ticket, determine whether or not it's a ticket, determine whether or not it's a feature or a bug.

feature or a bug. feature or a bug. From there, it'll do some code From there, it'll do some code From there, it'll do some code searching, searching, searching, provide all that context to the agent, provide all that context to the agent, provide all that context to the agent, and then come up with a plan of next and then come up with a plan of next and then come up with a plan of next steps. steps. steps. On the left-hand side here, this is a On the left-hand side here, this is a On the left-hand side here, this is a issue that one of my colleagues, uh issue that one of my colleagues, uh issue that one of my colleagues, uh smart engineer, had posted, I think, a smart engineer, had posted, I think, a smart engineer, had posted, I think, a month ago. month ago. month ago. Uh to summarize it, what had happened Uh to summarize it, what had happened Uh to summarize it, what had happened was, we had some pretty serious was, we had some pretty serious was, we had some pretty serious degradation in our agentic QA pipeline. degradation in our agentic QA pipeline. degradation in our agentic QA pipeline. Time to first character was taking 3 to Time to first character was taking 3 to Time to first character was taking 3 to 4 seconds when it should realistically 4 seconds when it should realistically 4 seconds when it should realistically be in the hundreds of milliseconds. be in the hundreds of milliseconds. be in the hundreds of milliseconds. So, let's see what happens when, you So, let's see what happens when, you So, let's see what happens when, you know, we put this through the system. So, as you'll see here, So, as you'll see here, I've set up the agent to go fetch the I've set up the agent to go fetch the I've set up the agent to go fetch the agent. agent. agent. I've given it the skills and tools to I've given it the skills and tools to I've given it the skills and tools to actually go and fetch the code, search actually go and fetch the code, search actually go and fetch the code, search the code, and query against that. the code, and query against that. the code, and query against that. That's being passed back to the agent, That's being passed back to the agent, That's being passed back to the agent, which is doing some reasoning against which is doing some reasoning against which is doing some reasoning against that right now. that right now. that right now. And then, wait a little bit. At this And then, wait a little bit. At this And then, wait a little bit. At this point, point, point, we've updated the Linear issue ticket. we've updated the Linear issue ticket. we've updated the Linear issue ticket. The recommendation here is to re-enable The recommendation here is to re-enable The recommendation here is to re-enable our async dispatch, which makes sense.

our async dispatch, which makes sense. our async dispatch, which makes sense. It allows us to run a lot more of our QE It allows us to run a lot more of our QE It allows us to run a lot more of our QE pipeline in parallel on a single pipeline in parallel on a single pipeline in parallel on a single machine. machine. machine. Sounds great, right? Unfortunately, Sounds great, right? Unfortunately, Sounds great, right? Unfortunately, uh this is wrong. uh this is wrong. uh this is wrong. This had actually caused an outage a few This had actually caused an outage a few This had actually caused an outage a few days ago in one of our uh support days ago in one of our uh support days ago in one of our uh support engineers had explicitly disabled this engineers had explicitly disabled this engineers had explicitly disabled this uh before this ticket was uh shown. uh before this ticket was uh shown. uh before this ticket was uh shown. So, where did things go wrong? Why was the where did things go wrong? Why was the uh uh uh you know, why did I get it wrong? you know, why did I get it wrong? you know, why did I get it wrong? The agent I had written, it didn't have The agent I had written, it didn't have The agent I had written, it didn't have a full picture. a full picture. a full picture. It was missing the context from the It was missing the context from the It was missing the context from the Slack discussion that happened after the Slack discussion that happened after the Slack discussion that happened after the issue where the engineers came together, issue where the engineers came together, issue where the engineers came together, uh went through the actual outage, what uh went through the actual outage, what uh went through the actual outage, what went wrong, what was the fix, and the went wrong, what was the fix, and the went wrong, what was the fix, and the next steps. next steps. next steps. It also was missing the postmortem uh It also was missing the postmortem uh It also was missing the postmortem uh linear ticket, which came as a result of linear ticket, which came as a result of linear ticket, which came as a result of that. that. that. In general, it had a narrow In general, it had a narrow In general, it had a narrow understanding of the problem. understanding of the problem. understanding of the problem. This concept of missing knowledge and This concept of missing knowledge and This concept of missing knowledge and intent that's stored across an intent that's stored across an intent that's stored across an organization and different systems is organization and different systems is organization and different systems is something that comes back and back something that comes back and back something that comes back and back again.

again. again. And since this was deployed as a And since this was deployed as a And since this was deployed as a background agent, this is going to make background agent, this is going to make background agent, this is going to make that mistake silently in the background, that mistake silently in the background, that mistake silently in the background, misinforming both my teammates and misinforming both my teammates and misinforming both my teammates and potentially other agents. potentially other agents. potentially other agents. So, I guess the next question is, why So, I guess the next question is, why So, I guess the next question is, why don't we run into this locally? You don't we run into this locally? You don't we run into this locally? You know, we all use agents locally, we know, we all use agents locally, we know, we all use agents locally, we don't necessarily run into these issues. don't necessarily run into these issues. don't necessarily run into these issues. Well, you, the human, the engineers, we Well, you, the human, the engineers, we Well, you, the human, the engineers, we currently act as that context layer. currently act as that context layer. currently act as that context layer. When working with an agent, you know, When working with an agent, you know, When working with an agent, you know, you're there to ask questions, catch any you're there to ask questions, catch any you're there to ask questions, catch any errors, and supply the missing facts on errors, and supply the missing facts on errors, and supply the missing facts on every single turn. every single turn. every single turn. A person knew why the code is the way it A person knew why the code is the way it A person knew why the code is the way it is, what broke last time, and what we've is, what broke last time, and what we've is, what broke last time, and what we've decided to do about it. decided to do about it. decided to do about it. The agent, though, it only has what's on The agent, though, it only has what's on The agent, though, it only has what's on the right-hand side, right? It has the right-hand side, right? It has the right-hand side, right? It has instructions, the tools and skills we instructions, the tools and skills we instructions, the tools and skills we specifically gave it, the code, as well specifically gave it, the code, as well specifically gave it, the code, as well as the ticket in front of it. as the ticket in front of it. as the ticket in front of it. When an agent is in the loop, well, When an agent is in the loop, well, When an agent is in the loop, well, sorry, when a human is in the loop with sorry, when a human is in the loop with sorry, when a human is in the loop with the agent, the agent, the agent, we're there to catch the steer. we're there to catch the steer. we're there to catch the steer. Ultimately, we're there to babysit the Ultimately, we're there to babysit the Ultimately, we're there to babysit the agent.

agent. agent. But as agents have gotten trivially easy But as agents have gotten trivially easy But as agents have gotten trivially easy to deploy as I showed earlier with Flu to deploy as I showed earlier with Flu to deploy as I showed earlier with Flu Cloudflare, Cloudflare, Cloudflare, the without the human in the loop, this the without the human in the loop, this the without the human in the loop, this issue becomes more and more prevalent. issue becomes more and more prevalent. issue becomes more and more prevalent. This missing context becomes a sign of This missing context becomes a sign of This missing context becomes a sign of failure. failure. failure. You know, You know, You know, all that intuition and knowledge that all that intuition and knowledge that all that intuition and knowledge that we've had as humans needs to be we've had as humans needs to be we've had as humans needs to be replaced. replaced. replaced. Something needs to carry the load. So, that thing, that's a context engine. So, that thing, that's a context engine. A context engine is a system that A context engine is a system that A context engine is a system that provides task-relevant information based provides task-relevant information based provides task-relevant information based on who you are and what matters. on who you are and what matters. on who you are and what matters. It also resolves all the conflicts It also resolves all the conflicts It also resolves all the conflicts across multiple data sets. across multiple data sets. across multiple data sets. It understands your access rules or the It understands your access rules or the It understands your access rules or the agent's access rules and only agent's access rules and only agent's access rules and only uh respects that and only provides uh respects that and only provides uh respects that and only provides information that's relevant. And most information that's relevant. And most information that's relevant. And most importantly, importantly, importantly, it delivers a synthesized it delivers a synthesized it delivers a synthesized understanding that an agent can act on, understanding that an agent can act on, understanding that an agent can act on, not just a list of documents that I have not just a list of documents that I have not just a list of documents that I have to reason upon itself. So, how does this context engine work? So, how does this context engine work? Well, let's take a step back. What does Well, let's take a step back. What does Well, let's take a step back. What does an agent actually need?

an agent actually need? an agent actually need? An agent needs Clearly, it needs context An agent needs Clearly, it needs context An agent needs Clearly, it needs context outside of just your source code. outside of just your source code. outside of just your source code. Think about everything that you need to Think about everything that you need to Think about everything that you need to work day-to-day. work day-to-day. work day-to-day. It's not just the code. It's, you know, It's not just the code. It's, you know, It's not just the code. It's, you know, the Slack discussions where decisions the Slack discussions where decisions the Slack discussions where decisions are made, are made, are made, the documentation where the documentation where the documentation where we show all the best practices. we show all the best practices. we show all the best practices. All that is important to your day-to-day All that is important to your day-to-day All that is important to your day-to-day process, and that's true for your agent process, and that's true for your agent process, and that's true for your agent as well. So, what we do here is we as well. So, what we do here is we as well. So, what we do here is we connect everything. The docs, code, connect everything. The docs, code, connect everything. The docs, code, tickets, tickets, tickets, conversations. We then build a model of conversations. We then build a model of conversations. We then build a model of your organization, of your system. And your organization, of your system. And your organization, of your system. And we piece how all these things work we piece how all these things work we piece how all these things work together and make it generally available together and make it generally available together and make it generally available to your agents. to your agents. to your agents. From that model, the agents are only From that model, the agents are only From that model, the agents are only provided a a of that data, which has provided a a of that data, which has provided a a of that data, which has been reconciled, ranked, and scoped to been reconciled, ranked, and scoped to been reconciled, ranked, and scoped to your permissions. your permissions. your permissions. Scattered context comes in, grounded Scattered context comes in, grounded Scattered context comes in, grounded context comes out. The obvious next question is, why can't The obvious next question is, why can't we just do this with MCP, right? You we just do this with MCP, right? You we just do this with MCP, right? You could connect a Slack MCP, a Linear MCP, could connect a Slack MCP, a Linear MCP, could connect a Slack MCP, a Linear MCP, a GitHub MCP, and with that, all that a GitHub MCP, and with that, all that a GitHub MCP, and with that, all that data is accessible.

data is accessible. data is accessible. MCP is great at access, but access isn't MCP is great at access, but access isn't MCP is great at access, but access isn't understanding. understanding. understanding. An MCP hands the agent the raw results, An MCP hands the agent the raw results, An MCP hands the agent the raw results, and you know, you're now dependent on and you know, you're now dependent on and you know, you're now dependent on that agent to actually that agent to actually that agent to actually decide what to believe in. decide what to believe in. decide what to believe in. You end up flooding the agent with You end up flooding the agent with You end up flooding the agent with irrelevant data, filling up the context irrelevant data, filling up the context irrelevant data, filling up the context window, and you know, overall context window, and you know, overall context window, and you know, overall context costs just go up. costs just go up. costs just go up. It also leaves the local agent to handle It also leaves the local agent to handle It also leaves the local agent to handle conflicts in data. You know, your Linear conflicts in data. You know, your Linear conflicts in data. You know, your Linear MCP and your Slack MCP may come back MCP and your Slack MCP may come back MCP and your Slack MCP may come back with different results. with different results. with different results. You're just leaving the agent to make You're just leaving the agent to make You're just leaving the agent to make that decision somewhat ad hoc at the that decision somewhat ad hoc at the that decision somewhat ad hoc at the moment. So, So, back to the original problem I had back to the original problem I had back to the original problem I had earlier. This is the same file, same earlier. This is the same file, same earlier. This is the same file, same engine, but now we've connected the engine, but now we've connected the engine, but now we've connected the context agent. context agent. context agent. Uh Uh Uh what we do here is is we're currently what we do here is is we're currently what we do here is is we're currently prompting Unblock to do some research on prompting Unblock to do some research on prompting Unblock to do some research on the ticket and provide that context to the ticket and provide that context to the ticket and provide that context to the agent. the agent. the agent. So, let's see that in action. So, here we go. Uh we're doing the very So, here we go. Uh we're doing the very similar thing. We're fetching the Linear similar thing. We're fetching the Linear similar thing. We're fetching the Linear ticket. But, you'll notice here that ticket. But, you'll notice here that ticket. But, you'll notice here that we're actually calling the Unblock we're actually calling the Unblock we're actually calling the Unblock context engine.

context engine. context engine. And what's done here is actually it's And what's done here is actually it's And what's done here is actually it's found the relevant Linear postmortem, as found the relevant Linear postmortem, as found the relevant Linear postmortem, as well as a Slack conversation where we've well as a Slack conversation where we've well as a Slack conversation where we've had the entire discussion between the had the entire discussion between the had the entire discussion between the engineering teams. engineering teams. engineering teams. And as part of that, we've returned a And as part of that, we've returned a And as part of that, we've returned a understanding, and that's now been understanding, and that's now been understanding, and that's now been provided to the agent as a summary. provided to the agent as a summary. provided to the agent as a summary. So, the agent no longer has to actually So, the agent no longer has to actually So, the agent no longer has to actually reason from those documents. reason from those documents. reason from those documents. Uh at this point, Uh at this point, Uh at this point, you'll notice here the agent now has been updated. the agent now has been updated. Uh the recommendation has gone from Uh the recommendation has gone from Uh the recommendation has gone from breaking and causing another issue to breaking and causing another issue to breaking and causing another issue to actually preventing a another outage. So, the example I've shown here is issue So, the example I've shown here is issue ticket management, but this context ticket management, but this context ticket management, but this context layer can actually go a lot further. layer can actually go a lot further. layer can actually go a lot further. Uh for example, coding. Uh for example, coding. Uh for example, coding. Everyone here does uh coding with uh Everyone here does uh coding with uh Everyone here does uh coding with uh cloud code or cortex. cloud code or cortex. cloud code or cortex. Using an Unblocked context engine to Using an Unblocked context engine to Using an Unblocked context engine to actually hydrate the agent plan goes a actually hydrate the agent plan goes a actually hydrate the agent plan goes a long way in terms of saving context and long way in terms of saving context and long way in terms of saving context and tokens.

tokens. tokens. Uh code review. Uh code review. Uh code review. It makes the PRs look as if they've been It makes the PRs look as if they've been It makes the PRs look as if they've been reviewed by an expert on your team. Who reviewed by an expert on your team. Who reviewed by an expert on your team. Who doesn't like that? doesn't like that? doesn't like that? As well as surfacing the correct answers As well as surfacing the correct answers As well as surfacing the correct answers to your customer success team as well as to your customer success team as well as to your customer success team as well as sales. sales. sales. In general, there are many instances In general, there are many instances In general, there are many instances where you might want an agent to have where you might want an agent to have where you might want an agent to have institutional and tribal knowledge of institutional and tribal knowledge of institutional and tribal knowledge of your organization. Just want to leave you on this. I think Just want to leave you on this. I think this quote encapsulates what we're this quote encapsulates what we're this quote encapsulates what we're trying to solve at Unblocked. The gap trying to solve at Unblocked. The gap trying to solve at Unblocked. The gap isn't intelligence, it's context. isn't intelligence, it's context. isn't intelligence, it's context. So, So, So, thank you. Uh thank you. Uh thank you. Uh I'll be at booth P16 along with the rest I'll be at booth P16 along with the rest I'll be at booth P16 along with the rest of my team if you guys have any of my team if you guys have any of my team if you guys have any questions. questions. questions. There will be additional breakout There will be additional breakout There will be additional breakout sessions later tomorrow, I believe, that sessions later tomorrow, I believe, that sessions later tomorrow, I believe, that goes a lot more in depth about actually goes a lot more in depth about actually goes a lot more in depth about actually how the context engine works and you how the context engine works and you how the context engine works and you know, how you can benefit from that. know, how you can benefit from that. know, how you can benefit from that. Cheers. Cheers. Cheers. >> [applause]