Multiplayer agentic engineering — Arjun Singh, Superconductor

>> All right. Hey everyone, I'm Arjun >> All right. Hey everyone, I'm Arjun Singh. Singh. Singh. Today I'm going to talk to you about Today I'm going to talk to you about Today I'm going to talk to you about multiplayer agentic engineering or how multiplayer agentic engineering or how multiplayer agentic engineering or how to enable your whole team and your best to enable your whole team and your best to enable your whole team and your best agents to work together. agents to work together. agents to work together. If you go to the talks or go around the If you go to the talks or go around the If you go to the talks or go around the expo, you're going to see that a lot of expo, you're going to see that a lot of expo, you're going to see that a lot of people are talking about putting the people are talking about putting the people are talking about putting the agents at the center of everything. agents at the center of everything. agents at the center of everything. Makes sense, they're really powerful, Makes sense, they're really powerful, Makes sense, they're really powerful, they're really cool. they're really cool. they're really cool. But you don't see a lot of people But you don't see a lot of people But you don't see a lot of people talking about the people. Like this is talking about the people. Like this is talking about the people. Like this is all for us to make us our lives better, all for us to make us our lives better, all for us to make us our lives better, our more our more productive, whatever. our more our more productive, whatever. our more our more productive, whatever. And so we're going to really focus on And so we're going to really focus on And so we're going to really focus on how the people fit into these agentic how the people fit into these agentic how the people fit into these agentic workflows. workflows. workflows. Just a little bit about us first. Just a little bit about us first. Just a little bit about us first. So our team has worked together building So our team has worked together building So our team has worked together building software for over a decade. Um my software for over a decade. Um my software for over a decade. Um my co-founder Sergey and I, we met in the co-founder Sergey and I, we met in the co-founder Sergey and I, we met in the PhD program at Berkeley. I worked on PhD program at Berkeley. I worked on PhD program at Berkeley. I worked on robotics, he worked on computer vision. robotics, he worked on computer vision. robotics, he worked on computer vision. And during that we co-founded a company And during that we co-founded a company And during that we co-founded a company called GradeScope. Some of you may have called GradeScope. Some of you may have called GradeScope. Some of you may have used it. It's used by millions of used it. It's used by millions of used it. It's used by millions of students worldwide at thousands of students worldwide at thousands of students worldwide at thousands of universities, helping instructors grade universities, helping instructors grade universities, helping instructors grade their students' work. their students' work. their students' work. And um pretty much the entire team And um pretty much the entire team And um pretty much the entire team working on Superconductor used to work working on Superconductor used to work working on Superconductor used to work together on GradeScope. And so we've had together on GradeScope. And so we've had together on GradeScope. And so we've had a team that's worked together a team that's worked together a team that's worked together productively from first user to productively from first user to productively from first user to acquisition, working on something new acquisition, working on something new acquisition, working on something new together again.

together again. together again. And I think it's kind of an interesting And I think it's kind of an interesting And I think it's kind of an interesting experiment because experiment because experiment because you know, over the past year we've very you know, over the past year we've very you know, over the past year we've very aggressively integrated agents in our aggressively integrated agents in our aggressively integrated agents in our workflows and we've kind of surfaced all workflows and we've kind of surfaced all workflows and we've kind of surfaced all the different bottlenecks and friction the different bottlenecks and friction the different bottlenecks and friction points that come up and how to do that points that come up and how to do that points that come up and how to do that productively and keep collaborating the productively and keep collaborating the productively and keep collaborating the way we used to but with the new power of way we used to but with the new power of way we used to but with the new power of agents. agents. agents. So today I'm going to talk to you about So today I'm going to talk to you about So today I'm going to talk to you about how, you know, we we the lessons we how, you know, we we the lessons we how, you know, we we the lessons we learned from kind of solving those learned from kind of solving those learned from kind of solving those friction points and solving those friction points and solving those friction points and solving those bottlenecks. bottlenecks. bottlenecks. And in the talk description I mentioned And in the talk description I mentioned And in the talk description I mentioned five lessons and I'm going to be an five lessons and I'm going to be an five lessons and I'm going to be an engineer and start from zero and add a engineer and start from zero and add a engineer and start from zero and add a sixth one in there. sixth one in there. sixth one in there. Um the first one I'm going to start with Um the first one I'm going to start with Um the first one I'm going to start with is just to be model and harness is just to be model and harness is just to be model and harness agnostic. agnostic. agnostic. So there's a few reasons for that. The So there's a few reasons for that. The So there's a few reasons for that. The best model and harness can change best model and harness can change best model and harness can change weekly. weekly. weekly. It could change cuz a new one comes out. It could change cuz a new one comes out. It could change cuz a new one comes out. It could change cuz the best one got It could change cuz the best one got It could change cuz the best one got taken away. taken away. taken away. Um things happen Um things happen Um things happen and you don't want that to disrupt your and you don't want that to disrupt your and you don't want that to disrupt your entire team's flow. entire team's flow. entire team's flow. The other thing is that open weight The other thing is that open weight The other thing is that open weight models are actually pretty good now. models are actually pretty good now. models are actually pretty good now. We've been really happy with GLM 5.2. We've been really happy with GLM 5.2. We've been really happy with GLM 5.2. They're much cheaper. They're much cheaper. They're much cheaper. Um and you want to be able to kind of Um and you want to be able to kind of Um and you want to be able to kind of explore with them and integrate them explore with them and integrate them explore with them and integrate them without again having to change your without again having to change your without again having to change your entire workflow.

entire workflow. entire workflow. The last thing I'll mention on this is The last thing I'll mention on this is The last thing I'll mention on this is that the incentives of the people that the incentives of the people that the incentives of the people selling you tokens aren't really aligned selling you tokens aren't really aligned selling you tokens aren't really aligned with yours. You're here for a reason. with yours. You're here for a reason. with yours. You're here for a reason. You're you're you're So you're doing You're you're you're So you're doing You're you're you're So you're doing things for a reason. You're trying to things for a reason. You're trying to things for a reason. You're trying to make your customers' lives better, make make your customers' lives better, make make your customers' lives better, make your product better, delight your your product better, delight your your product better, delight your customers. customers. customers. They want to sell you more tokens. And They want to sell you more tokens. And They want to sell you more tokens. And you might be happy to pay for as many you might be happy to pay for as many you might be happy to pay for as many tokens as it takes, but you don't want tokens as it takes, but you don't want tokens as it takes, but you don't want to pay for more than that. to pay for more than that. to pay for more than that. And so again, kind of being able to And so again, kind of being able to And so again, kind of being able to switch between things lets you stay in switch between things lets you stay in switch between things lets you stay in control of all of that. control of all of that. control of all of that. And so, um you know, as I go through the And so, um you know, as I go through the And so, um you know, as I go through the talk, talk, talk, I'll mention a couple of places where I'll mention a couple of places where I'll mention a couple of places where our product makes it easy for us, but our product makes it easy for us, but our product makes it easy for us, but whether you use us or not, I'm just whether you use us or not, I'm just whether you use us or not, I'm just going to leave things with you that I going to leave things with you that I going to leave things with you that I think are really important um for you to think are really important um for you to think are really important um for you to be able to work collaboratively be able to work collaboratively be able to work collaboratively effectively. effectively. effectively. So, the next one is to turn every human So, the next one is to turn every human So, the next one is to turn every human interface into an agent and human interface into an agent and human interface into an agent and human interface. interface. interface. So, you know, typically when people are So, you know, typically when people are So, you know, typically when people are working with coding agents, they're on working with coding agents, they're on working with coding agents, they're on their laptop, their laptop, their laptop, kind of stuck on that laptop. Nobody kind of stuck on that laptop. Nobody kind of stuck on that laptop. Nobody else can talk to that agent. else can talk to that agent. else can talk to that agent. So, the first place people go to kind of So, the first place people go to kind of So, the first place people go to kind of expose more interfaces for them is expose more interfaces for them is expose more interfaces for them is Slack. Cloud has a Slack bot. Coda has a Slack. Cloud has a Slack bot. Coda has a Slack. Cloud has a Slack bot. Coda has a Slack bot. We have a Slack bot. It's Slack bot. We have a Slack bot. It's Slack bot. We have a Slack bot. It's really cool. You can say, "Hey, Coda really cool. You can say, "Hey, Coda really cool. You can say, "Hey, Coda bot, do XYZ." It does it. Somebody else bot, do XYZ." It does it. Somebody else bot, do XYZ." It does it. Somebody else can talk to it.

can talk to it. can talk to it. But it's not enough. But it's not enough. But it's not enough. Cuz now we've taken it from trapped on Cuz now we've taken it from trapped on Cuz now we've taken it from trapped on somebody's laptop to kind of trapped in somebody's laptop to kind of trapped in somebody's laptop to kind of trapped in Slack. Slack. Slack. And a lot of work happened in Slack, so And a lot of work happened in Slack, so And a lot of work happened in Slack, so that's better than than than nothing, that's better than than than nothing, that's better than than than nothing, but certainly not all work happens in but certainly not all work happens in but certainly not all work happens in Slack. Slack. Slack. So, what we really wanted was to be able So, what we really wanted was to be able So, what we really wanted was to be able to work with the same session from every to work with the same session from every to work with the same session from every relevant interface. Could be Slack, relevant interface. Could be Slack, relevant interface. Could be Slack, could be our app, could be GitHub, could could be our app, could be GitHub, could could be our app, could be GitHub, could be elsewhere. be elsewhere. be elsewhere. And so, one possible flow is you start And so, one possible flow is you start And so, one possible flow is you start and collaborate on a session in Slack. and collaborate on a session in Slack. and collaborate on a session in Slack. And then maybe you continue in a kind of And then maybe you continue in a kind of And then maybe you continue in a kind of more engineer-focused environment in the more engineer-focused environment in the more engineer-focused environment in the desktop app or the mobile app. desktop app or the mobile app. desktop app or the mobile app. And then, you can finish it up in And then, you can finish it up in And then, you can finish it up in GitHub. GitHub. GitHub. And the important thing here is the And the important thing here is the And the important thing here is the exact same agent session. So, it's like exact same agent session. So, it's like exact same agent session. So, it's like the agent didn't forget what you did in the agent didn't forget what you did in the agent didn't forget what you did in one place in Slack when you go and talk one place in Slack when you go and talk one place in Slack when you go and talk to it from GitHub, it's the same to it from GitHub, it's the same to it from GitHub, it's the same session. It's got the same context. And the second lesson builds on top of And the second lesson builds on top of that, which is to make the agent work that, which is to make the agent work that, which is to make the agent work visible and collaborative across the visible and collaborative across the visible and collaborative across the team. And so, obviously Slack makes it team. And so, obviously Slack makes it team. And so, obviously Slack makes it more collaborative. more collaborative. more collaborative. Um Um Um But here, we've got that kind of app But here, we've got that kind of app But here, we've got that kind of app view. And you know, Sergey made this view. And you know, Sergey made this view. And you know, Sergey made this ticket. I've been talking to the same ticket. I've been talking to the same ticket. I've been talking to the same ticket. Our growth person hopped in as ticket. Our growth person hopped in as ticket. Our growth person hopped in as well. So, you can kind of see at the top well. So, you can kind of see at the top well. So, you can kind of see at the top here all the different people that here all the different people that here all the different people that interacted with this. So, I can see interacted with this. So, I can see interacted with this. So, I can see who's getting notified about this who's getting notified about this who's getting notified about this session, who's seen it. Um that's session, who's seen it. Um that's session, who's seen it. Um that's especially important when you have work especially important when you have work especially important when you have work triggered by non-technical people.

triggered by non-technical people. triggered by non-technical people. Right? So, it's like, you know, our Right? So, it's like, you know, our Right? So, it's like, you know, our customer support person created a customer support person created a customer support person created a ticket. It's working really well. I want ticket. It's working really well. I want ticket. It's working really well. I want to understand like has this been vetted to understand like has this been vetted to understand like has this been vetted by an engineer or not? You can kind of by an engineer or not? You can kind of by an engineer or not? You can kind of see who's involved really easily. see who's involved really easily. see who's involved really easily. And then, if I'm reviewing something, I And then, if I'm reviewing something, I And then, if I'm reviewing something, I can just pop in and say like, "Hey, why can just pop in and say like, "Hey, why can just pop in and say like, "Hey, why did you do it this way?" And again, did you do it this way?" And again, did you do it this way?" And again, because it's the same agent session, because it's the same agent session, because it's the same agent session, I don't need to wait for Sergey to kind I don't need to wait for Sergey to kind I don't need to wait for Sergey to kind of get my notification on GitHub and of get my notification on GitHub and of get my notification on GitHub and respond to me. The answer to the respond to me. The answer to the respond to me. The answer to the question is almost certainly in this question is almost certainly in this question is almost certainly in this thread. thread. thread. I don't also don't want to read the I don't also don't want to read the I don't also don't want to read the entire thread. So, I can just ask ask entire thread. So, I can just ask ask entire thread. So, I can just ask ask the agent. the agent. the agent. Or Or Or how we most often kind of make the work how we most often kind of make the work how we most often kind of make the work visible is with artifacts. visible is with artifacts. visible is with artifacts. So, it doesn't matter where the work So, it doesn't matter where the work So, it doesn't matter where the work started or where it's finishing, the started or where it's finishing, the started or where it's finishing, the agent can show you the work it's doing agent can show you the work it's doing agent can show you the work it's doing as screenshot or video or other. as screenshot or video or other. as screenshot or video or other. And you can see it from everywhere. And you can see it from everywhere. And you can see it from everywhere. So, again, you don't have to worry about So, again, you don't have to worry about So, again, you don't have to worry about like, "Oh, where is that thing? I got to like, "Oh, where is that thing? I got to like, "Oh, where is that thing? I got to go to GitHub to see the image or got to go to GitHub to see the image or got to go to GitHub to see the image or got to go to Slack to see the image." It's just go to Slack to see the image." It's just go to Slack to see the image." It's just everywhere. Work is visible everywhere. everywhere. Work is visible everywhere. everywhere. Work is visible everywhere. Collaborate from anywhere. The third lesson The third lesson >> [clears throat] >> [clears throat] >> [clears throat] >> um I'm going to talk about here is to >> um I'm going to talk about here is to >> um I'm going to talk about here is to turn every external signal into code turn every external signal into code turn every external signal into code that your team can quickly evaluate.

that your team can quickly evaluate. that your team can quickly evaluate. And I I to show this live, but the Wi-Fi And I I to show this live, but the Wi-Fi And I I to show this live, but the Wi-Fi is not quite there. So, I'm going to is not quite there. So, I'm going to is not quite there. So, I'm going to show you something from yesterday. show you something from yesterday. show you something from yesterday. But, what do I mean by external signal? But, what do I mean by external signal? But, what do I mean by external signal? So, it could be Slack conversation, So, it could be Slack conversation, So, it could be Slack conversation, could be a meeting you have with a could be a meeting you have with a could be a meeting you have with a customer, an onboarding call, a sales customer, an onboarding call, a sales customer, an onboarding call, a sales call. Could be an internal team meeting, call. Could be an internal team meeting, call. Could be an internal team meeting, could be something from Sentry or bug could be something from Sentry or bug could be something from Sentry or bug tracker, a bug report from a customer, tracker, a bug report from a customer, tracker, a bug report from a customer, an email, feature request. an email, feature request. an email, feature request. And right now what's happening is like And right now what's happening is like And right now what's happening is like all that stuff is already exists. It's all that stuff is already exists. It's all that stuff is already exists. It's in all those different systems. in all those different systems. in all those different systems. People hook them together with MCPs. So, People hook them together with MCPs. So, People hook them together with MCPs. So, now your coding agent can check the now your coding agent can check the now your coding agent can check the email or check Notion or whatever it email or check Notion or whatever it email or check Notion or whatever it might be. But, like might be. But, like might be. But, like how do you how how does it know what to how do you how how does it know what to how do you how how does it know what to work on, right? It's like it's still work on, right? It's like it's still work on, right? It's like it's still kind of stuck everywhere. And so, like kind of stuck everywhere. And so, like kind of stuck everywhere. And so, like some humans are involved in like kind of some humans are involved in like kind of some humans are involved in like kind of taking stuff from one place and taking stuff from one place and taking stuff from one place and telling it solve email number 48 or telling it solve email number 48 or telling it solve email number 48 or ticket number 6,000. ticket number 6,000. ticket number 6,000. But, that's still a lot of coordination. But, that's still a lot of coordination. But, that's still a lot of coordination. And so, what we do is we we have several And so, what we do is we we have several And so, what we do is we we have several different ways to automatically ingest different ways to automatically ingest different ways to automatically ingest these signals, these signals, these signals, prioritize what to what to do with it prioritize what to what to do with it prioritize what to what to do with it and and kind of act on them. and and kind of act on them. and and kind of act on them. And my favorite one, the most fun one, And my favorite one, the most fun one, And my favorite one, the most fun one, is what we call our meeting bot.

is what we call our meeting bot. is what we call our meeting bot. And so, I'm going to switch switch over And so, I'm going to switch switch over And so, I'm going to switch switch over to my browser here for a second. to my browser here for a second. to my browser here for a second. And uh And uh And uh Okay. Okay. Okay. So, we've got a booth at the expo and we So, we've got a booth at the expo and we So, we've got a booth at the expo and we had the meeting bot running all day had the meeting bot running all day had the meeting bot running all day yesterday. So, this is a 4-hour meeting yesterday. So, this is a 4-hour meeting yesterday. So, this is a 4-hour meeting of a Google Meet. You just kind of of a Google Meet. You just kind of of a Google Meet. You just kind of invite the bot to Meet or Zoom or Teams invite the bot to Meet or Zoom or Teams invite the bot to Meet or Zoom or Teams or whatever it might be. or whatever it might be. or whatever it might be. And it listens all day. And it listens all day. And it listens all day. And it created all sorts of stuff as it And it created all sorts of stuff as it And it created all sorts of stuff as it was listening. was listening. was listening. If it finds existing work, it'll link to If it finds existing work, it'll link to If it finds existing work, it'll link to it. All right? So, it's not going to it. All right? So, it's not going to it. All right? So, it's not going to just like create new work if it's just like create new work if it's just like create new work if it's something already working on. Some of something already working on. Some of something already working on. Some of this is, you know, people testing the this is, you know, people testing the this is, you know, people testing the meeting bot out and telling it to do meeting bot out and telling it to do meeting bot out and telling it to do some weird things, um or interesting some weird things, um or interesting some weird things, um or interesting things, or just creative ideas. But, a things, or just creative ideas. But, a things, or just creative ideas. But, a lot of it's actually just like really lot of it's actually just like really lot of it's actually just like really good ideas that come out of people good ideas that come out of people good ideas that come out of people looking at what we're doing, asking looking at what we're doing, asking looking at what we're doing, asking questions, having new ideas of what to questions, having new ideas of what to questions, having new ideas of what to do with it. do with it. do with it. And so, it's kind of nice cuz the last And so, it's kind of nice cuz the last And so, it's kind of nice cuz the last idea that was here was someone saying, idea that was here was someone saying, idea that was here was someone saying, "Hey, like when I work with coding "Hey, like when I work with coding "Hey, like when I work with coding agents, I want to make sure that agents, I want to make sure that agents, I want to make sure that the agent has clear criteria to evaluate the agent has clear criteria to evaluate the agent has clear criteria to evaluate whether it did a good job on the work whether it did a good job on the work whether it did a good job on the work before it tells me that it's done. before it tells me that it's done. before it tells me that it's done. And um so they had that idea, the bot And um so they had that idea, the bot And um so they had that idea, the bot just picked up on it. None of us did just picked up on it. None of us did just picked up on it. None of us did anything manually.

anything manually. anything manually. It created this ticket It created this ticket It created this ticket and started working on it. and started working on it. and started working on it. And then I was able to just say, "Hey, And then I was able to just say, "Hey, And then I was able to just say, "Hey, take a screenshot of what what you did." take a screenshot of what what you did." take a screenshot of what what you did." And here's that screenshot. And here's that screenshot. And here's that screenshot. And it kind of modified our ticket form And it kind of modified our ticket form And it kind of modified our ticket form to add these two new fields of to add these two new fields of to add these two new fields of acceptance criteria. acceptance criteria. acceptance criteria. Now, I'm going to Am I going to ship Now, I'm going to Am I going to ship Now, I'm going to Am I going to ship this one exactly how it is? Like, no, this one exactly how it is? Like, no, this one exactly how it is? Like, no, probably not. But it's a new idea, it's probably not. But it's a new idea, it's probably not. But it's a new idea, it's concrete, I can play with it, I can go concrete, I can play with it, I can go concrete, I can play with it, I can go and actually like use the live preview and actually like use the live preview and actually like use the live preview and like see if this improves and like see if this improves and like see if this improves performance. performance. performance. And so it takes like this you know, you And so it takes like this you know, you And so it takes like this you know, you know, your hundreds or thousands of know, your hundreds or thousands of know, your hundreds or thousands of ideas that are everywhere and it helps ideas that are everywhere and it helps ideas that are everywhere and it helps you kind of move with the speed of what you kind of move with the speed of what you kind of move with the speed of what your customers are asking you for and your customers are asking you for and your customers are asking you for and what they're thinking. what they're thinking. what they're thinking. And um And um And um it's really fun because every time we it's really fun because every time we it's really fun because every time we have an onboarding your customer call or have an onboarding your customer call or have an onboarding your customer call or or team meeting, we almost always have or team meeting, we almost always have or team meeting, we almost always have dozens of new ideas that are prototyped, dozens of new ideas that are prototyped, dozens of new ideas that are prototyped, but more importantly, at least a few but more importantly, at least a few but more importantly, at least a few shippable PRs with a very minimal shippable PRs with a very minimal shippable PRs with a very minimal intervention. intervention. intervention. So we talk, stuff comes out, we look at So we talk, stuff comes out, we look at So we talk, stuff comes out, we look at it, we ship it. it, we ship it. it, we ship it. It's so much fun. So the next thing I'm going to mention So the next thing I'm going to mention is that, you know, the these three is that, you know, the these three is that, you know, the these three things that I've talked to you about things that I've talked to you about things that I've talked to you about really rely really rely really rely on having your workflow, on having your workflow, on having your workflow, your code base, your project set up to your code base, your project set up to your code base, your project set up to work in an isolated cloud environment.

work in an isolated cloud environment. work in an isolated cloud environment. So that way the agents aren't trapped on So that way the agents aren't trapped on So that way the agents aren't trapped on an individual's machine. So there's several reasons why this is So there's several reasons why this is important. important. important. So So So the first one is to eliminate what some the first one is to eliminate what some the first one is to eliminate what some people are calling lid anxiety. You want people are calling lid anxiety. You want people are calling lid anxiety. You want to be able to close your laptop. You've to be able to close your laptop. You've to be able to close your laptop. You've probably seen people running around the probably seen people running around the probably seen people running around the office with their laptops open while office with their laptops open while office with their laptops open while stuff is working or at the airport or stuff is working or at the airport or stuff is working or at the airport or you know, there's some posts on Twitter you know, there's some posts on Twitter you know, there's some posts on Twitter or whatever about um you know, like or whatever about um you know, like or whatever about um you know, like people having their laptop tethered to people having their laptop tethered to people having their laptop tethered to their phone in their cars as they're their phone in their cars as they're their phone in their cars as they're driving home. driving home. driving home. This was actually probably the impetus This was actually probably the impetus This was actually probably the impetus for for me and for a few people on our for for me and for a few people on our for for me and for a few people on our team to even start working on this. team to even start working on this. team to even start working on this. You You last year I started working on You You last year I started working on You You last year I started working on Cloud code with cloud code a lot. Cloud code with cloud code a lot. Cloud code with cloud code a lot. I had a I think at the time like I had a I think at the time like I had a I think at the time like 6-month-old. I like didn't want to like 6-month-old. I like didn't want to like 6-month-old. I like didn't want to like be tied to my laptop or have that be tied to my laptop or have that be tied to my laptop or have that stress. I was like, I I don't I I don't stress. I was like, I I don't I I don't stress. I was like, I I don't I I don't ever want to think about whether I can ever want to think about whether I can ever want to think about whether I can like step away from a laptop or not. like step away from a laptop or not. like step away from a laptop or not. And so, we moved everything to the And so, we moved everything to the And so, we moved everything to the cloud. Things are working always. cloud. Things are working always. cloud. Things are working always. It eliminated that problem for us that It eliminated that problem for us that It eliminated that problem for us that people have been talking about for the people have been talking about for the people have been talking about for the past year. past year. past year. It's really helpful. It's important to It's really helpful. It's important to It's really helpful. It's important to me. me. me. But I don't think that's the most But I don't think that's the most But I don't think that's the most important reason to do this. I think the important reason to do this. I think the important reason to do this. I think the most important reason to do this most important reason to do this most important reason to do this actually was was touched on in the actually was was touched on in the actually was was touched on in the previous talk, if you were here for it.

previous talk, if you were here for it. previous talk, if you were here for it. I think you should only give your access I think you should only give your access I think you should only give your access give your agents access to only what give your agents access to only what give your agents access to only what they need. they need. they need. Right? So, if you think about what's Right? So, if you think about what's Right? So, if you think about what's happening, you have a bunch of happening, you have a bunch of happening, you have a bunch of developers with these agents running on developers with these agents running on developers with these agents running on their laptop. their laptop. their laptop. Their laptops, unless you have like Their laptops, unless you have like Their laptops, unless you have like impeccable hygiene, probably have a impeccable hygiene, probably have a impeccable hygiene, probably have a bunch of stuff on it that you don't want bunch of stuff on it that you don't want bunch of stuff on it that you don't want the LLMs or agents to have access to. the LLMs or agents to have access to. the LLMs or agents to have access to. And yeah, like everybody's working on And yeah, like everybody's working on And yeah, like everybody's working on these sandboxes and approval flows. And these sandboxes and approval flows. And these sandboxes and approval flows. And so, you but but but really you're in one so, you but but but really you're in one so, you but but but really you're in one of two camps. You're either approving a of two camps. You're either approving a of two camps. You're either approving a bunch of stuff bunch of stuff bunch of stuff or you're hoping that your auto approval or you're hoping that your auto approval or you're hoping that your auto approval flow or your YOLO mode or whatever is flow or your YOLO mode or whatever is flow or your YOLO mode or whatever is configured properly and your sandbox is configured properly and your sandbox is configured properly and your sandbox is configured properly and doesn't read a configured properly and doesn't read a configured properly and doesn't read a bunch of stuff on your laptop that it bunch of stuff on your laptop that it bunch of stuff on your laptop that it shouldn't have. shouldn't have. shouldn't have. And, you know, as as the previous talk And, you know, as as the previous talk And, you know, as as the previous talk mentioned, like these agents are getting mentioned, like these agents are getting mentioned, like these agents are getting more autonomous. They're getting really more autonomous. They're getting really more autonomous. They're getting really resourceful. They They're trying to resourceful. They They're trying to resourceful. They They're trying to please you and do what you said. And so, please you and do what you said. And so, please you and do what you said. And so, when you say, "Hey, you know, wipe this when you say, "Hey, you know, wipe this when you say, "Hey, you know, wipe this agent database." and it finds a token on agent database." and it finds a token on agent database." and it finds a token on your laptop that it can use and it your laptop that it can use and it your laptop that it can use and it thinks it's working with staging, but thinks it's working with staging, but thinks it's working with staging, but actually it's production and now it just actually it's production and now it just actually it's production and now it just deleted everything. I'm not trying to deleted everything. I'm not trying to deleted everything. I'm not trying to say this is happening constantly, but it say this is happening constantly, but it say this is happening constantly, but it still happens. And for us, the peace of still happens. And for us, the peace of still happens. And for us, the peace of mind of just like letting anybody run mind of just like letting anybody run mind of just like letting anybody run with these experiments and ideas and with these experiments and ideas and with these experiments and ideas and prototypes and and and real code prototypes and and and real code prototypes and and and real code without having to worry about this is is without having to worry about this is is without having to worry about this is is is is really worthwhile.

is is really worthwhile. is is really worthwhile. To go one step further on that, um it's To go one step further on that, um it's To go one step further on that, um it's not just, "Hey, make sure they don't not just, "Hey, make sure they don't not just, "Hey, make sure they don't have the credentials that they shouldn't have the credentials that they shouldn't have the credentials that they shouldn't have." have." have." It's also make sure they can't It's also make sure they can't It's also make sure they can't exfiltrate your code or your projects or exfiltrate your code or your projects or exfiltrate your code or your projects or your secrets or your content to your secrets or your content to your secrets or your content to somewhere they shouldn't be able to. somewhere they shouldn't be able to. somewhere they shouldn't be able to. And so you have a configurable network And so you have a configurable network And so you have a configurable network sandbox and you say, "Look, these are sandbox and you say, "Look, these are sandbox and you say, "Look, these are the places you're allowed to access, the places you're allowed to access, the places you're allowed to access, these are the ones you can't access." these are the ones you can't access." these are the ones you can't access." And anytime it tries to access something And anytime it tries to access something And anytime it tries to access something that it shouldn't, it just pops up and that it shouldn't, it just pops up and that it shouldn't, it just pops up and says, "Hey, says, "Hey, says, "Hey, tried to access something. Do you want tried to access something. Do you want tried to access something. Do you want to give it access? Maybe you're trying to give it access? Maybe you're trying to give it access? Maybe you're trying to integrate a new vendor and you need to integrate a new vendor and you need to integrate a new vendor and you need documentation." documentation." documentation." And you can do it on a per ticket basis And you can do it on a per ticket basis And you can do it on a per ticket basis or for the whole project. And so again, or for the whole project. And so again, or for the whole project. And so again, that peace of mind of like people can do that peace of mind of like people can do that peace of mind of like people can do things. things. things. If they need new access, it's easy to If they need new access, it's easy to If they need new access, it's easy to grant it. And um grant it. And um grant it. And um we're not going to leak a bunch of we're not going to leak a bunch of we're not going to leak a bunch of important data important data important data uh uh uh by uh running agents in yellow mode. And the last thing I'll mention about And the last thing I'll mention about that is that that is that that is that this is the key for allowing your this is the key for allowing your this is the key for allowing your non-technical team members to trigger non-technical team members to trigger non-technical team members to trigger real work. Right? Your non-technical real work. Right? Your non-technical real work. Right? Your non-technical people don't have development people don't have development people don't have development environments set up on their computers.

environments set up on their computers. environments set up on their computers. But we've gotten our support people or But we've gotten our support people or But we've gotten our support people or growth people to actually be growth people to actually be growth people to actually be meaningfully impacting the product meaningfully impacting the product meaningfully impacting the product by just talking to the users, seeing by just talking to the users, seeing by just talking to the users, seeing bugs, experiencing themselves, and just bugs, experiencing themselves, and just bugs, experiencing themselves, and just go to Slack or the rabbit cell and say, go to Slack or the rabbit cell and say, go to Slack or the rabbit cell and say, "Hey, fix this." "Hey, fix this." "Hey, fix this." They fix it. Screenshots are shown. They fix it. Screenshots are shown. They fix it. Screenshots are shown. Engineer gets it, gets merged. Engineer gets it, gets merged. Engineer gets it, gets merged. Without that, they'd have to put it in Without that, they'd have to put it in Without that, they'd have to put it in linear and linear would eventually pick linear and linear would eventually pick linear and linear would eventually pick it up and a PM would triage it or it up and a PM would triage it or it up and a PM would triage it or whatever. whatever. whatever. None of that here. You just ask for it None of that here. You just ask for it None of that here. You just ask for it and it's done. Now, the reason people didn't do this Now, the reason people didn't do this you know, up until somewhat recently, you know, up until somewhat recently, you know, up until somewhat recently, like this was really painful. Getting like this was really painful. Getting like this was really painful. Getting your full thing set up in this like kind your full thing set up in this like kind your full thing set up in this like kind of sandbox environment used to be of sandbox environment used to be of sandbox environment used to be really, really painful. really, really painful. really, really painful. But agents have gotten better. But agents have gotten better. But agents have gotten better. We have our own environment setup We have our own environment setup We have our own environment setup assistant that kind of takes your assistant that kind of takes your assistant that kind of takes your project and gets it to work in one of project and gets it to work in one of project and gets it to work in one of these sandboxes. these sandboxes. these sandboxes. But honestly, whether you use this or But honestly, whether you use this or But honestly, whether you use this or not, I highly recommend you get your not, I highly recommend you get your not, I highly recommend you get your project working this way and you can project working this way and you can project working this way and you can just get Cloud Coder or CodeX to do this just get Cloud Coder or CodeX to do this just get Cloud Coder or CodeX to do this for you. You don't have to use us, but for you. You don't have to use us, but for you. You don't have to use us, but you know, we think it's the best way. And the last lesson is to benchmark And the last lesson is to benchmark agents on your code base.

agents on your code base. agents on your code base. So, um the way we do this, we select So, um the way we do this, we select So, um the way we do this, we select pull requests that represent great pull requests that represent great pull requests that represent great engineering work. It could be agent engineering work. It could be agent engineering work. It could be agent created, could be human created, could created, could be human created, could created, could be human created, could be a hybrid, doesn't matter. be a hybrid, doesn't matter. be a hybrid, doesn't matter. You pick the agents you want to use and You pick the agents you want to use and You pick the agents you want to use and benchmark. benchmark. benchmark. And then you get a quality versus cost And then you get a quality versus cost And then you get a quality versus cost and time breakdown on your code base. and time breakdown on your code base. and time breakdown on your code base. Now, why do you want to do this? There's Now, why do you want to do this? There's Now, why do you want to do this? There's There's many reasons, but one is that if There's many reasons, but one is that if There's many reasons, but one is that if you're kind of going off the public you're kind of going off the public you're kind of going off the public benchmarks, we bench or terminal bench benchmarks, we bench or terminal bench benchmarks, we bench or terminal bench or other stuff like those tasks may have or other stuff like those tasks may have or other stuff like those tasks may have absolutely nothing to do with your task. absolutely nothing to do with your task. absolutely nothing to do with your task. Like swe bench is all in Python, we're Like swe bench is all in Python, we're Like swe bench is all in Python, we're Ruby on Rails. Ruby on Rails. Ruby on Rails. It is not the case that the benchmarks It is not the case that the benchmarks It is not the case that the benchmarks are identical for them. There's trends are identical for them. There's trends are identical for them. There's trends that do compare, but the results can be that do compare, but the results can be that do compare, but the results can be very, very different. very, very different. very, very different. And I'm going to swap over to my browser And I'm going to swap over to my browser And I'm going to swap over to my browser one more time here. So, this is These are results on our So, this is These are results on our code base of all these different code base of all these different code base of all these different harnesses. This one here is quality harnesses. This one here is quality harnesses. This one here is quality versus cost. versus cost. versus cost. This one here is quality versus time. This one here is quality versus time. This one here is quality versus time. I'll start with this one. I'll start with this one. I'll start with this one. You can see some trends here. Right, you You can see some trends here. Right, you You can see some trends here. Right, you can see that the Anthropic can see that the Anthropic can see that the Anthropic agents have just been consistently agents have just been consistently agents have just been consistently getting better, but not really any getting better, but not really any getting better, but not really any faster.

faster. faster. The Codex agents and cursor are actually The Codex agents and cursor are actually The Codex agents and cursor are actually pretty fast and quite good. pretty fast and quite good. pretty fast and quite good. The open stuff has been getting better The open stuff has been getting better The open stuff has been getting better and better over time, but they're kind and better over time, but they're kind and better over time, but they're kind of slow. This is for our code base of slow. This is for our code base of slow. This is for our code base again. I'm not trying to make any again. I'm not trying to make any again. I'm not trying to make any general claims here. general claims here. general claims here. By cost, By cost, By cost, the Anthropic stuff is clearly just so the Anthropic stuff is clearly just so the Anthropic stuff is clearly just so much more expensive for us. much more expensive for us. much more expensive for us. And the Codex stuff has been cheaper for And the Codex stuff has been cheaper for And the Codex stuff has been cheaper for us. us. us. And so this causes us to change our And so this causes us to change our And so this causes us to change our behavior. We still use the different behavior. We still use the different behavior. We still use the different models. There's different use cases for models. There's different use cases for models. There's different use cases for them. We like the variety. We still use them. We like the variety. We still use them. We like the variety. We still use all these things. But when we saw these all these things. But when we saw these all these things. But when we saw these results, they kind of matched our vibe results, they kind of matched our vibe results, they kind of matched our vibe check. We wanted to kind of like have check. We wanted to kind of like have check. We wanted to kind of like have hard data, too. We switched our default hard data, too. We switched our default hard data, too. We switched our default to Codex at that time. to Codex at that time. to Codex at that time. And Fiable came out, and it was great. And Fiable came out, and it was great. And Fiable came out, and it was great. Kind of switched our default to that for Kind of switched our default to that for Kind of switched our default to that for like the few days we had it, and then it like the few days we had it, and then it like the few days we had it, and then it went away and switched back to Codex. went away and switched back to Codex. went away and switched back to Codex. But the most important thing is like But the most important thing is like But the most important thing is like because we're agnostic, like none of because we're agnostic, like none of because we're agnostic, like none of that had any meaningful disruption on that had any meaningful disruption on that had any meaningful disruption on our work. Like we're able to just kind our work. Like we're able to just kind our work. Like we're able to just kind of switch back and forth really easily. of switch back and forth really easily. of switch back and forth really easily. So, the next day something new comes So, the next day something new comes So, the next day something new comes out, out, out, see if it's good, see if it's good, see if it's good, and go. and go. and go. And the last thing I want to mention And the last thing I want to mention And the last thing I want to mention around that is like around that is like around that is like I don't know if this resonates with you I don't know if this resonates with you I don't know if this resonates with you all, but all, but all, but I have a lot of friends that like, I have a lot of friends that like, I have a lot of friends that like, "Okay, and I heard MiniMax is good. I "Okay, and I heard MiniMax is good. I "Okay, and I heard MiniMax is good. I heard, you know, GLM is good, and Kimmy heard, you know, GLM is good, and Kimmy heard, you know, GLM is good, and Kimmy K2 is good, but like haven't had the K2 is good, but like haven't had the K2 is good, but like haven't had the time to try it out, and everyone keeps time to try it out, and everyone keeps time to try it out, and everyone keeps telling me I need to cuz it's so much telling me I need to cuz it's so much telling me I need to cuz it's so much better and faster and cheaper." And you better and faster and cheaper." And you better and faster and cheaper." And you kind of have that anxiety for a little kind of have that anxiety for a little kind of have that anxiety for a little while, and then like finally you take while, and then like finally you take while, and then like finally you take the two hours to try it, and it's like, the two hours to try it, and it's like, the two hours to try it, and it's like, "Oh, actually like "Oh, actually like "Oh, actually like didn't really work for us. So, like didn't really work for us. So, like didn't really work for us. So, like what's, you know, I just wasted those what's, you know, I just wasted those what's, you know, I just wasted those two hours." It kind of eliminates that.

two hours." It kind of eliminates that. two hours." It kind of eliminates that. It helps you kind of stay on the cutting It helps you kind of stay on the cutting It helps you kind of stay on the cutting edge really like seamlessly. Let me go back. Let me go back. So, what that kind of turned into us for So, what that kind of turned into us for So, what that kind of turned into us for us is, you know, us is, you know, us is, you know, essentially 100% like 99.9% of our pull essentially 100% like 99.9% of our pull essentially 100% like 99.9% of our pull requests are like heavily agent requests are like heavily agent requests are like heavily agent generated. generated. generated. We know that We know that We know that quality and reliability and security are quality and reliability and security are quality and reliability and security are really important, so we still have really important, so we still have really important, so we still have humans look at everything. We have humans look at everything. We have humans look at everything. We have agents help with it all, but agents help with it all, but agents help with it all, but everything's human reviewed. everything's human reviewed. everything's human reviewed. Um Um Um you know, for our our our our relatively you know, for our our our our relatively you know, for our our our our relatively small team, we had 1 and 1/2 billion small team, we had 1 and 1/2 billion small team, we had 1 and 1/2 billion tokens over the past month. tokens over the past month. tokens over the past month. And you can kind of see what we're And you can kind of see what we're And you can kind of see what we're saying about Claude here. It's a little saying about Claude here. It's a little saying about Claude here. It's a little small, so I apologize, but we had 3,300 small, so I apologize, but we had 3,300 small, so I apologize, but we had 3,300 Claude code runs that cost $10,000 in Claude code runs that cost $10,000 in Claude code runs that cost $10,000 in tokens daily. We have plans, so we tokens daily. We have plans, so we tokens daily. We have plans, so we didn't spend $10,000 on it. didn't spend $10,000 on it. didn't spend $10,000 on it. And Codex had four times as many And Codex had four times as many And Codex had four times as many sessions, sessions, sessions, and it was cheaper overall. and it was cheaper overall. and it was cheaper overall. And so again, the vast majority of our And so again, the vast majority of our And so again, the vast majority of our work currently is merged through Codex. work currently is merged through Codex. work currently is merged through Codex. We still use the other models. More and We still use the other models. More and We still use the other models. More and more is happening through GLM 5.2. more is happening through GLM 5.2. more is happening through GLM 5.2. Going to invest in that. Going to invest in that. Going to invest in that. And And And the one thing that we're really excited the one thing that we're really excited the one thing that we're really excited to do going forward with this to do going forward with this to do going forward with this benchmarking is automatically like like benchmarking is automatically like like benchmarking is automatically like like you've probably heard about people, you you've probably heard about people, you you've probably heard about people, you know, routing tasks to the right models know, routing tasks to the right models know, routing tasks to the right models and all that, but how do how does like and all that, but how do how does like and all that, but how do how does like some third party know what to route for some third party know what to route for some third party know what to route for your code base? Like this is a way that your code base? Like this is a way that your code base? Like this is a way that you can know what's going to work best you can know what's going to work best you can know what's going to work best for which task for your for your for which task for your for your for which task for your for your project, and we're going to kind of project, and we're going to kind of project, and we're going to kind of automatically routing that for you.

So, I'm going to leave you with a few So, I'm going to leave you with a few recommendations. recommendations. recommendations. So, first, um get your code base and So, first, um get your code base and So, first, um get your code base and agents working in a sandbox. It unlocks agents working in a sandbox. It unlocks agents working in a sandbox. It unlocks a lot of different things, a lot of a lot of different things, a lot of a lot of different things, a lot of different workflows, everything I've different workflows, everything I've different workflows, everything I've talked about and more. talked about and more. talked about and more. Second, Second, Second, integrate agents into the relevant human integrate agents into the relevant human integrate agents into the relevant human interfaces so your team and your agents interfaces so your team and your agents interfaces so your team and your agents can work together and don't have to like can work together and don't have to like can work together and don't have to like context switch and copy context back and context switch and copy context back and context switch and copy context back and forth. forth. forth. Obviously, we think Superagent is the Obviously, we think Superagent is the Obviously, we think Superagent is the best way to do it, but plenty of people best way to do it, but plenty of people best way to do it, but plenty of people are home rolling things, are home rolling things, are home rolling things, hacking things together. hacking things together. hacking things together. Figure out how to make this happen Figure out how to make this happen Figure out how to make this happen because if not, the friction is just because if not, the friction is just because if not, the friction is just really high. really high. really high. And lastly, And lastly, And lastly, find a way to benchmark and become model find a way to benchmark and become model find a way to benchmark and become model agnostic so you're not tied to anybody agnostic so you're not tied to anybody agnostic so you're not tied to anybody and you can just constantly stay at that and you can just constantly stay at that and you can just constantly stay at that right part on the frontier of cost, right part on the frontier of cost, right part on the frontier of cost, speed, quality. So, thank you so much. Um, we've got a So, thank you so much. Um, we've got a booth in the expo. Please feel free to booth in the expo. Please feel free to booth in the expo. Please feel free to come by. You can sign up at come by. You can sign up at come by. You can sign up at superagent.com or you can email me with superagent.com or you can email me with superagent.com or you can email me with any questions at [email protected]. any questions at [email protected]. any questions at [email protected]. Um, I will be out in the back as well Um, I will be out in the back as well Um, I will be out in the back as well for any questions. Thanks so much. for any questions. Thanks so much. for any questions. Thanks so much. >> [applause] >> One one one last thing, um, >> One one one last thing, um, if, uh, you know, at the booth we're if, uh, you know, at the booth we're if, uh, you know, at the booth we're mentioning we're giving away a MacBook mentioning we're giving away a MacBook mentioning we're giving away a MacBook Neo. If you were here cuz you signed up Neo. If you were here cuz you signed up Neo. If you were here cuz you signed up through for that, just meet us outside through for that, just meet us outside through for that, just meet us outside and we will announce the winner. Thank and we will announce the winner. Thank and we will announce the winner. Thank you.