ACP: The Universal Remote Control for AI Agents — Alex Hancock, Block

>> Hey everybody. >> Hey everybody. Um Um Um my name's Alex Hancock. Today I'm going my name's Alex Hancock. Today I'm going my name's Alex Hancock. Today I'm going to talk about a universal remote control to talk about a universal remote control to talk about a universal remote control for AI. for AI. for AI. And before I start, I just want to say And before I start, I just want to say And before I start, I just want to say the previous speaker said that MCP the previous speaker said that MCP the previous speaker said that MCP client maintainers haven't implemented client maintainers haven't implemented client maintainers haven't implemented support for tasks because they're smart. support for tasks because they're smart. support for tasks because they're smart. I'm an MCP client maintainer. I can tell I'm an MCP client maintainer. I can tell I'm an MCP client maintainer. I can tell you it's just because I'm lazy. I you it's just because I'm lazy. I you it's just because I'm lazy. I haven't done it. haven't done it. haven't done it. Um Um Um Okay. So, a little bit about me before Okay. So, a little bit about me before Okay. So, a little bit about me before we start. I am a software engineer at we start. I am a software engineer at we start. I am a software engineer at Block, which is the parent company of Block, which is the parent company of Block, which is the parent company of Cash App and Square and Title. We have a Cash App and Square and Title. We have a Cash App and Square and Title. We have a few different things going on now. Um few different things going on now. Um few different things going on now. Um and I've worked there for a long time. I and I've worked there for a long time. I and I've worked there for a long time. I worked on Square product stuff and Cash worked on Square product stuff and Cash worked on Square product stuff and Cash App stuff, but I've been doing open App stuff, but I've been doing open App stuff, but I've been doing open source AI for the last couple years. source AI for the last couple years. source AI for the last couple years. Specifically, I work on this open source Specifically, I work on this open source Specifically, I work on this open source harness project called Goose, which harness project called Goose, which harness project called Goose, which started as an internal project at Block. started as an internal project at Block. started as an internal project at Block. Yeah, some Goose fans out there. And Yeah, some Goose fans out there. And Yeah, some Goose fans out there. And then uh yeah, we open sourced it and we then uh yeah, we open sourced it and we then uh yeah, we open sourced it and we donated it to the Linux Foundation. So, donated it to the Linux Foundation. So, donated it to the Linux Foundation. So, now the IP is there, but we still lots now the IP is there, but we still lots now the IP is there, but we still lots of us from Block still work on it. of us from Block still work on it. of us from Block still work on it. Um I'm also a maintainer of MCP, the Um I'm also a maintainer of MCP, the Um I'm also a maintainer of MCP, the Model Context Protocol. I work on the Model Context Protocol. I work on the Model Context Protocol. I work on the Rust SDK for that project. And more Rust SDK for that project. And more Rust SDK for that project. And more recently, I've also started some work on recently, I've also started some work on recently, I've also started some work on ACP, the Agent Client Protocol, which is ACP, the Agent Client Protocol, which is ACP, the Agent Client Protocol, which is what I'm going to talk about today.

what I'm going to talk about today. what I'm going to talk about today. So, I think I think we have an issue So, I think I think we have an issue So, I think I think we have an issue with harnesses that I want to I want to with harnesses that I want to I want to with harnesses that I want to I want to try to put to you all today, try to put to you all today, try to put to you all today, uh propose to you all today as a problem uh propose to you all today as a problem uh propose to you all today as a problem and then and then recommend a a and then and then recommend a a and then and then recommend a a solution. solution. solution. So, what I've been noticing recently So, what I've been noticing recently So, what I've been noticing recently is that we've got lots of great is that we've got lots of great is that we've got lots of great harnesses out there, right? There are harnesses out there, right? There are harnesses out there, right? There are ones from the labs, there are one from ones from the labs, there are one from ones from the labs, there are one from ones from different companies, there's ones from different companies, there's ones from different companies, there's lots of open standards-based ones. lots of open standards-based ones. lots of open standards-based ones. Um but I noticed that the interface to Um but I noticed that the interface to Um but I noticed that the interface to them them them is often custom or bespoke. And in in is often custom or bespoke. And in in is often custom or bespoke. And in in the worst case, it's like you might have the worst case, it's like you might have the worst case, it's like you might have some harnesses where there's literally some harnesses where there's literally some harnesses where there's literally only one client application you can use only one client application you can use only one client application you can use to control that harness, right? And I to control that harness, right? And I to control that harness, right? And I think this has a couple issues with it, think this has a couple issues with it, think this has a couple issues with it, but the analogy that I'll make with the but the analogy that I'll make with the but the analogy that I'll make with the web is it would be like if you had to web is it would be like if you had to web is it would be like if you had to use one browser or a use one browser or a use one browser or a one given protocol to connect to a to one given protocol to connect to a to one given protocol to connect to a to every website, right? That just wouldn't every website, right? That just wouldn't every website, right? That just wouldn't work. You wouldn't have something like work. You wouldn't have something like work. You wouldn't have something like the open web if if that were the reality the open web if if that were the reality the open web if if that were the reality with browsers. And so I think we can do with browsers. And so I think we can do with browsers. And so I think we can do better. better. better. And the thing about standards by finding And the thing about standards by finding And the thing about standards by finding a standard and the thing about standards a standard and the thing about standards a standard and the thing about standards is that they create ecosystems and is that they create ecosystems and is that they create ecosystems and markets.

markets. markets. And I would argue that in the agentic AI And I would argue that in the agentic AI And I would argue that in the agentic AI space, we have a good standard for the space, we have a good standard for the space, we have a good standard for the agent going out and doing things, right? agent going out and doing things, right? agent going out and doing things, right? Calling tools, taking actions in other Calling tools, taking actions in other Calling tools, taking actions in other systems, reading resources, reading systems, reading resources, reading systems, reading resources, reading data. We've all benefited as a community data. We've all benefited as a community data. We've all benefited as a community from having MCP. from having MCP. from having MCP. Right? And the most powerful thing about Right? And the most powerful thing about Right? And the most powerful thing about MCP is not anything about MCP itself, MCP is not anything about MCP itself, MCP is not anything about MCP itself, but it's that everyone uses MCP. but it's that everyone uses MCP. but it's that everyone uses MCP. And that's why we have, you know, And that's why we have, you know, And that's why we have, you know, thousands or tens of thousands of thousands or tens of thousands of thousands or tens of thousands of servers around the world and all the servers around the world and all the servers around the world and all the agents can connect to them and go and do agents can connect to them and go and do agents can connect to them and go and do things in those other systems. things in those other systems. things in those other systems. I would say that we don't yet have a I would say that we don't yet have a I would say that we don't yet have a good solution or a standard for client good solution or a standard for client good solution or a standard for client software to tell agents what to do. software to tell agents what to do. software to tell agents what to do. Giving it tasks, telling it what to work Giving it tasks, telling it what to work Giving it tasks, telling it what to work on and getting updates. on and getting updates. on and getting updates. And so And so And so I'm going to put forward an option today I'm going to put forward an option today I'm going to put forward an option today that I think is a good option that that that I think is a good option that that that I think is a good option that that we on our team have been working on and we on our team have been working on and we on our team have been working on and we think is a good a good solution in we think is a good a good solution in we think is a good a good solution in the open standards space. the open standards space. the open standards space. And this is ACP, so agent client And this is ACP, so agent client And this is ACP, so agent client protocol protocol protocol is the name of this project and it came is the name of this project and it came is the name of this project and it came from the editor companies. It came from from the editor companies. It came from from the editor companies. It came from like if you've used the Zed text editor like if you've used the Zed text editor like if you've used the Zed text editor or you've used any of JetBrains or you've used any of JetBrains or you've used any of JetBrains products, the Zed folks and the products, the Zed folks and the products, the Zed folks and the JetBrains JetBrains folks teamed up and JetBrains JetBrains folks teamed up and JetBrains JetBrains folks teamed up and proposed a standard for proposed a standard for proposed a standard for um clients to be able to control um clients to be able to control um clients to be able to control harnesses. And it makes sense if you put harnesses. And it makes sense if you put harnesses. And it makes sense if you put yourself in their shoes, right? What yourself in their shoes, right? What yourself in their shoes, right? What they wanted to be able to do is write a they wanted to be able to do is write a they wanted to be able to do is write a single high quality client single high quality client single high quality client implementation in an editor, maybe in implementation in an editor, maybe in implementation in an editor, maybe in Zed or in IntelliJ or something like Zed or in IntelliJ or something like Zed or in IntelliJ or something like that, and be able to control any harness

that, and be able to control any harness that, and be able to control any harness by with that single client by with that single client by with that single client implementation, sending tasks, getting implementation, sending tasks, getting implementation, sending tasks, getting results back, results back, results back, seeing what files are being edited, etc. seeing what files are being edited, etc. seeing what files are being edited, etc. It makes a ton of sense if you put It makes a ton of sense if you put It makes a ton of sense if you put yourself in their shoes, right? But we yourself in their shoes, right? But we yourself in their shoes, right? But we saw this on the Goose team, and we think saw this on the Goose team, and we think saw this on the Goose team, and we think that there is a much broader utility that there is a much broader utility that there is a much broader utility than just editors, right? So it's a than just editors, right? So it's a than just editors, right? So it's a it's it's relatively neutral and it it's it's relatively neutral and it it's it's relatively neutral and it doesn't have many editor specific doesn't have many editor specific doesn't have many editor specific features. And so we think that this can features. And so we think that this can features. And so we think that this can can go to a be spread to a wider range can go to a be spread to a wider range can go to a be spread to a wider range of client software. of client software. of client software. To go into a little bit more depth about To go into a little bit more depth about To go into a little bit more depth about ACP's design and and what you can do ACP's design and and what you can do ACP's design and and what you can do with it, with it, with it, it lets you establish connections it lets you establish connections it lets you establish connections between clients and agent harnesses that between clients and agent harnesses that between clients and agent harnesses that have a given have a given have a given a set of capabilities associated with a set of capabilities associated with a set of capabilities associated with the the connection, and then you can the the connection, and then you can the the connection, and then you can make sessions. Within sessions, you can make sessions. Within sessions, you can make sessions. Within sessions, you can send user messages, the things that a send user messages, the things that a send user messages, the things that a user is maybe typing into the app or user is maybe typing into the app or user is maybe typing into the app or that the client software wants to send. that the client software wants to send. that the client software wants to send. The agent can then respond to those with The agent can then respond to those with The agent can then respond to those with text, text, text, uh more images or audio, text, etc., or uh more images or audio, text, etc., or uh more images or audio, text, etc., or updates about what's going on. So like, updates about what's going on. So like, updates about what's going on. So like, if a tool is called, it can send a tool if a tool is called, it can send a tool if a tool is called, it can send a tool call notification and explain what tool call notification and explain what tool call notification and explain what tool was called and what the metadata was.

was called and what the metadata was. was called and what the metadata was. Um and it can also send things like Um and it can also send things like Um and it can also send things like permission requests, so that if the permission requests, so that if the permission requests, so that if the client software needs to show the user, client software needs to show the user, client software needs to show the user, you know, "Should I do this tool call? you know, "Should I do this tool call? you know, "Should I do this tool call? Yes or no?" It can go over this uh Yes or no?" It can go over this uh Yes or no?" It can go over this uh this protocol. And it's it's pretty this protocol. And it's it's pretty this protocol. And it's it's pretty simple in its design. Uh it uses JSON simple in its design. Uh it uses JSON simple in its design. Uh it uses JSON RPC messages, and the thing we like RPC messages, and the thing we like RPC messages, and the thing we like about it most is that it's extensible as about it most is that it's extensible as about it most is that it's extensible as well, so you're not limited to just well, so you're not limited to just well, so you're not limited to just what's in the vanilla protocol. You can what's in the vanilla protocol. You can what's in the vanilla protocol. You can add custom methods. So the add custom methods. So the add custom methods. So the the convention is you put an underscore, the convention is you put an underscore, the convention is you put an underscore, and then you start to put your custom and then you start to put your custom and then you start to put your custom methods. And the thing I like about this methods. And the thing I like about this methods. And the thing I like about this is that if enough harness projects or is that if enough harness projects or is that if enough harness projects or client projects adopt this, client projects adopt this, client projects adopt this, we can start to see what we're all doing we can start to see what we're all doing we can start to see what we're all doing that's the same, that's the same, that's the same, right? Like if the Codex team has some right? Like if the Codex team has some right? Like if the Codex team has some custom methods, the Goose team has some custom methods, the Goose team has some custom methods, the Goose team has some custom methods, custom methods, custom methods, uh the client team has some custom uh the client team has some custom uh the client team has some custom methods, whoever, we can see what methods, whoever, we can see what methods, whoever, we can see what emerges in the in the ecosystem and what emerges in the in the ecosystem and what emerges in the in the ecosystem and what makes sense to get on a standards track makes sense to get on a standards track makes sense to get on a standards track and bring into the protocol itself so and bring into the protocol itself so and bring into the protocol itself so that this is sort of shaped by usage and that this is sort of shaped by usage and that this is sort of shaped by usage and shaped by the community. shaped by the community. shaped by the community. Um I'm going to do a demo Um I'm going to do a demo Um I'm going to do a demo of a standard IO version of this. So, of a standard IO version of this. So, of a standard IO version of this. So, I'm going to open Zed and I just have a I'm going to open Zed and I just have a I'm going to open Zed and I just have a really simple project here really simple project here really simple project here where I'll say, "Tell me about this where I'll say, "Tell me about this where I'll say, "Tell me about this project." And so, this is a single HTML project." And so, this is a single HTML project." And so, this is a single HTML file.

file. file. So, you can see I was able to type my So, you can see I was able to type my So, you can see I was able to type my query into Zed and this is the agent in query into Zed and this is the agent in query into Zed and this is the agent in play here is Goose, so it's using play here is Goose, so it's using play here is Goose, so it's using Goose's ACP interface. Goose's ACP interface. Goose's ACP interface. And it's you can see it's like sending And it's you can see it's like sending And it's you can see it's like sending text back, it's sending tool call text back, it's sending tool call text back, it's sending tool call information back information back information back um about what it read and what it did um about what it read and what it did um about what it read and what it did and then it found, you know, that it's a and then it found, you know, that it's a and then it found, you know, that it's a single HTML file and then explained it. single HTML file and then explained it. single HTML file and then explained it. And I'll do another And I'll do another And I'll do another I'll do another one. This is one from a I'll do another one. This is one from a I'll do another one. This is one from a company called Poolside AI. company called Poolside AI. company called Poolside AI. Uh Uh Uh I'll say, "Tell me about this project." I'll say, "Tell me about this project." I'll say, "Tell me about this project." In the same project. And so, this is a In the same project. And so, this is a In the same project. And so, this is a terminal-based client getting exactly terminal-based client getting exactly terminal-based client getting exactly the same experience from the same agent, the same experience from the same agent, the same experience from the same agent, one implementation on the harness side, one implementation on the harness side, one implementation on the harness side, and you can now use any client. Right? and you can now use any client. Right? and you can now use any client. Right? And so, you can see it did the same And so, you can see it did the same And so, you can see it did the same thing. It showed me thing. It showed me thing. It showed me some text results back, it showed a tool some text results back, it showed a tool some text results back, it showed a tool call, and then it showed a It's it's call, and then it showed a It's it's call, and then it showed a It's it's streaming in a summary. streaming in a summary. streaming in a summary. Um Um Um So, that's a basic demo showing two So, that's a basic demo showing two So, that's a basic demo showing two clients talking to the same agent clients talking to the same agent clients talking to the same agent uh over standard IO locally in this uh over standard IO locally in this uh over standard IO locally in this case. case. case. But local obviously isn't enough, right? But local obviously isn't enough, right? But local obviously isn't enough, right? If you want this to take off, you have If you want this to take off, you have If you want this to take off, you have to be able to do remote as well. Agents to be able to do remote as well. Agents to be able to do remote as well. Agents are going to be running in the cloud.

are going to be running in the cloud. are going to be running in the cloud. And so, when we came to this project, we And so, when we came to this project, we And so, when we came to this project, we saw that it did not have remote support saw that it did not have remote support saw that it did not have remote support yet. So, we specified an HTTP transport. yet. So, we specified an HTTP transport. yet. So, we specified an HTTP transport. There's an HTTP version and there's a There's an HTTP version and there's a There's an HTTP version and there's a websocket upgrade. And so now, the websocket upgrade. And so now, the websocket upgrade. And so now, the messages are the same, the protocol messages are the same, the protocol messages are the same, the protocol semantics are the same, but there's a semantics are the same, but there's a semantics are the same, but there's a new transport that is just landing now new transport that is just landing now new transport that is just landing now that enables remote. that enables remote. that enables remote. And And And the way we think about this on the Goose the way we think about this on the Goose the way we think about this on the Goose team, the agentic stack, is there's sort team, the agentic stack, is there's sort team, the agentic stack, is there's sort of these four important components, of these four important components, of these four important components, right? You have the client, right? You have the client, right? You have the client, which is like the app that the user is which is like the app that the user is which is like the app that the user is using or a headless app running using or a headless app running using or a headless app running somewhere on a machine. There's the somewhere on a machine. There's the somewhere on a machine. There's the harness, which is the program that harness, which is the program that harness, which is the program that implements the tool calling loop. implements the tool calling loop. implements the tool calling loop. There are the tools themselves. This There are the tools themselves. This There are the tools themselves. This often MCP. And then there's the model, often MCP. And then there's the model, often MCP. And then there's the model, right? And if you do a remote transport right? And if you do a remote transport right? And if you do a remote transport for the agent client protocol, for the agent client protocol, for the agent client protocol, and MCP has remote transport for tool and MCP has remote transport for tool and MCP has remote transport for tool calling, calling, calling, and the models have kind of all had and the models have kind of all had and the models have kind of all had remote endpoints like responses APIs for remote endpoints like responses APIs for remote endpoints like responses APIs for a long time. Now you have the a long time. Now you have the a long time. Now you have the flexibility to move all of these four flexibility to move all of these four flexibility to move all of these four components around. They could all be on components around. They could all be on components around. They could all be on the same machine. The harness could be the same machine. The harness could be the same machine. The harness could be on a different machine than the client.

on a different machine than the client. on a different machine than the client. Uh the model could be the only thing Uh the model could be the only thing Uh the model could be the only thing that's remote. The tools could be the that's remote. The tools could be the that's remote. The tools could be the only thing that's remote. Uh aligning on only thing that's remote. Uh aligning on only thing that's remote. Uh aligning on standards and making sure that they have standards and making sure that they have standards and making sure that they have good transport stories is what's going good transport stories is what's going good transport stories is what's going to let us move all the pieces of this to let us move all the pieces of this to let us move all the pieces of this agentic stack around. agentic stack around. agentic stack around. And I can show a quick demo of this as And I can show a quick demo of this as And I can show a quick demo of this as well. well. well. So, this is a a client just to show how So, this is a a client just to show how So, this is a a client just to show how easy it is to create clients for this. I easy it is to create clients for this. I easy it is to create clients for this. I just live-coded this, you know, last just live-coded this, you know, last just live-coded this, you know, last night. And I'll say, "Write a poem." So, night. And I'll say, "Write a poem." So, night. And I'll say, "Write a poem." So, this is again connecting to that same this is again connecting to that same this is again connecting to that same process on my machine. process on my machine. process on my machine. Uh in this case I'm running it over the Uh in this case I'm running it over the Uh in this case I'm running it over the network, but it's on my machine. It's network, but it's on my machine. It's network, but it's on my machine. It's connecting and sending Goose connecting and sending Goose connecting and sending Goose instructions for what to do uh remotely. instructions for what to do uh remotely. instructions for what to do uh remotely. So, this could be in a container, could So, this could be in a container, could So, this could be in a container, could be up in the cloud, but the messages are be up in the cloud, but the messages are be up in the cloud, but the messages are the same and the library you use is the the same and the library you use is the the same and the library you use is the same. So, you can just switch between same. So, you can just switch between same. So, you can just switch between local and remote very, very easily. local and remote very, very easily. local and remote very, very easily. Um Um Um So, if you want to get plugged into this So, if you want to get plugged into this So, if you want to get plugged into this ecosystem, start experimenting with ecosystem, start experimenting with ecosystem, start experimenting with support, either making your own clients support, either making your own clients support, either making your own clients or adding stuff to harnesses, this is a or adding stuff to harnesses, this is a or adding stuff to harnesses, this is a this will link you to the agent client this will link you to the agent client this will link you to the agent client protocol site for protocol site for protocol site for how to get started. There's a number of how to get started. There's a number of how to get started. There's a number of clients and and agent servers are out clients and and agent servers are out clients and and agent servers are out there. This ranges from editors, desktop there. This ranges from editors, desktop there. This ranges from editors, desktop applications, mobile applications, applications, mobile applications, applications, mobile applications, terminal based things, like there's a terminal based things, like there's a terminal based things, like there's a proliferation.

proliferation. proliferation. And And And I I think the use cases are are are I I think the use cases are are are I I think the use cases are are are potentially huge, right? If we if we get potentially huge, right? If we if we get potentially huge, right? If we if we get some interoperability going here because some interoperability going here because some interoperability going here because you can have people can make personal you can have people can make personal you can have people can make personal clients that's exactly how you want it clients that's exactly how you want it clients that's exactly how you want it orchestrating your agents. You could orchestrating your agents. You could orchestrating your agents. You could have sort of clients created for certain have sort of clients created for certain have sort of clients created for certain business domains or an individual business domains or an individual business domains or an individual company or a set of clients from a company or a set of clients from a company or a set of clients from a company you could customize like a white company you could customize like a white company you could customize like a white label label label client have it work with all the client have it work with all the client have it work with all the harnesses. And I also think if we make a harnesses. And I also think if we make a harnesses. And I also think if we make a new category here, new category here, new category here, we're going to see quality of the we're going to see quality of the we're going to see quality of the clients go up, right? Because any clients go up, right? Because any clients go up, right? Because any anytime you get an ecosystem or a anytime you get an ecosystem or a anytime you get an ecosystem or a marketplace going and there's many marketplace going and there's many marketplace going and there's many options, options, options, users can vote with their feet if users can vote with their feet if users can vote with their feet if clients aren't meeting their needs and clients aren't meeting their needs and clients aren't meeting their needs and so people will start to compete on the so people will start to compete on the so people will start to compete on the quality of the user experience and and quality of the user experience and and quality of the user experience and and like overall I think this should drive like overall I think this should drive like overall I think this should drive up uh up uh up uh the user experience of using AI. the user experience of using AI. the user experience of using AI. That's what I've got today. Thank you That's what I've got today. Thank you That's what I've got today. Thank you very much. And if you want to chat with very much. And if you want to chat with very much. And if you want to chat with me, find me after or send me an email. me, find me after or send me an email. me, find me after or send me an email. Um happy to get you plugged into this Um happy to get you plugged into this Um happy to get you plugged into this work. work. work. Thank you.