Building on the Codex Harness — Dominik Kundel, OpenAI

Well, friends. Well, friends. Hello everyone. Um, I want to Hello everyone. Um, I want to Hello everyone. Um, I want to start by asking you to start by asking you to start by asking you to quickly raise your hands. quickly raise your hands. quickly raise your hands. How many of you How many of you How many of you have created your own have created your own have created your own agent shell? agent shell? agent shell? Good. And how many of you Good. And how many of you , um, feel that , um, feel that , um, feel that creating a shell creating a shell creating a shell for an agent is too for an agent is too for an agent is too scary? Okay, cool. scary? Okay, cool. scary? Okay, cool. So, for the next 18 So, for the next 18 So, for the next 18 minutes or minutes or minutes or so, I want so, I want so, I want to talk to you about to talk to you about to talk to you about how you can how you can how you can build on top of the build on top of the build on top of the Codex shell and Codex shell and Codex shell and hopefully make hopefully make hopefully make the process of creating the process of creating the process of creating your own your own agent interfaces a little agent interfaces a little easier and less easier and less easier and less complicated. And complicated. And complicated. And hopefully hopefully hopefully a little more interesting too. Um, a little more interesting too. Um, a little more interesting too. Um, my name is Dom. I my name is Dom. I my name is Dom. I work at OpenAI on work at OpenAI on work at OpenAI on developer experience developer experience , specifically for Codex. Um, and , specifically for Codex. Um, and , specifically for Codex. Um, and another request for a another request for a another request for a show of hands. I promise, show of hands. I promise, show of hands. I promise, I will give you a break I will give you a break . Um, but how many of you . Um, but how many of you . Um, but how many of you have heard of the have heard of the have heard of the Codex application server? Codex application server? Codex application server? Okay, that was a bit of a Okay, that was a bit of a Okay, that was a bit of a trick question trick question trick question because if you were at because if you were at because if you were at yesterday's presentation, yesterday's presentation, yesterday's presentation, you all should have raised your you all should have raised your you all should have raised your hands. Um, otherwise you hands. Um, otherwise you hands. Um, otherwise you probably wouldn't have listened to probably wouldn't have listened to probably wouldn't have listened to Roman. Um, but we Roman. Um, but we Roman. Um, but we briefly mentioned briefly mentioned briefly mentioned this yesterday, but we didn't this yesterday, but we didn't this yesterday, but we didn't explain what an explain what an explain what an application server is. application server is. application server is. Um, so the Um, so the Um, so the application server is application server is application server is actually a protocol actually a protocol actually a protocol that wraps the that wraps the that wraps the Codex shell. Um, if you've ever Codex shell. Um, if you've ever wondered about wondered about wondered about something like the something like the Agent Client Protocol (ACP), it's a very Agent Client Protocol (ACP), it's a very similar concept, where similar concept, where similar concept, where we essentially have a we essentially have a we essentially have a protocol that protocol that protocol that allows you to build allows you to build allows you to build directly on top of the directly on top of the directly on top of the shell and shell and shell and integrate deeply into it.

integrate deeply into it. integrate deeply into it. And we'll talk a little bit And we'll talk a little bit And we'll talk a little bit in this talk about in this talk about in this talk about what the differences are between an what the differences are between an what the differences are between an application server application server application server and an ACP. It is important and an ACP. It is important and an ACP. It is important to remember that both the to remember that both the to remember that both the Codex shell and the Codex shell and the Codex shell and the application server itself are application server itself are application server itself are open source. open source. open source. So, we'll look at So, we'll look at So, we'll look at a few different a few different a few different ways to interact with ways to interact with ways to interact with it. Um, but more it. Um, but more it. Um, but more importantly, we'll importantly, we'll be diving into how be diving into how some of this stuff some of this stuff some of this stuff works under the hood tomorrow afternoon, works under the hood tomorrow afternoon, works under the hood tomorrow afternoon, but you but you but you can always ask Codex can always ask Codex can always ask Codex to read to read to read the repository and the repository and the repository and dive into it dive into it dive into it yourself. This is an yourself. This is an yourself. This is an Apache 2 license, so Apache 2 license, so Apache 2 license, so feel free to fork, feel free to fork, feel free to fork, modify, and modify, and modify, and have have have fun. You fun. You fun. You can even can even can even connect it to connect it to connect it to any other any other model provider as long as they have an model provider as long as they have an API compatible with the API compatible with the API compatible with the Responses API. We even Responses API. We even Responses API. We even have convenient have convenient have convenient settings for settings for settings for easy connection easy connection easy connection to LM Studio or Ollama. As for to LM Studio or Ollama. As for to LM Studio or Ollama. As for how it works: the how it works: the application server protocol is a application server protocol is a JSON RPC-style protocol JSON RPC-style protocol that we actually that we actually that we actually use to use to use to run the Codex app, the run the Codex app, the run the Codex app, the IDE extension, our IDE extension, our IDE extension, our VS Code extension, VS Code extension, VS Code extension, i.e. all of our i.e. all of our i.e. all of our own interfaces as well own interfaces as well own interfaces as well as third-party ones.

as third-party ones. as third-party ones. So, if you've So, if you've ever used ever used ever used Codex through Xcode, JetBrains IDE, Codex through Xcode, JetBrains IDE, Codex through Xcode, JetBrains IDE, or even some or even some or even some open open open source projects like Theos T3 Code source projects like Theos T3 Code source projects like Theos T3 Code or Remote X, they all or Remote X, they all or Remote X, they all run on the same run on the same run on the same foundation, on the foundation, on the foundation, on the same same same application server. Earlier application server. Earlier application server. Earlier this year, we even this year, we even this year, we even used it to used it to used it to implement Codex in Cloud implement Codex in Cloud implement Codex in Cloud Code, leveraging the Code, leveraging the Code, leveraging the same same same application server. So, if application server. So, if application server. So, if you're still in Cloud Code and you you're still in Cloud Code and you you're still in Cloud Code and you want to want to want to use use use Codex there to Codex there to Codex there to review your code review your code review your code or, say, or, say, or, say, pass it pass it pass it a task to a task to a task to do, you can do, you can do, you can use this use this use this plugin, which behind plugin, which behind plugin, which behind the scenes the scenes the scenes uses the uses the uses the same same same application server, the same application server, the same application server, the same foundation. If you're foundation. If you're foundation. If you're familiar with familiar with familiar with things like ACP, this should things like ACP, this should things like ACP, this should seem pretty seem pretty seem pretty familiar. In general, familiar. In general, familiar. In general, the way an the way an the way an application server works is that application server works is that application server works is that we we we exchange exchange exchange messages between messages between messages between the client, i.e. the client, i.e. the client, i.e. the application you are the application you are the application you are building, and the server building, and the server , i.e. the , i.e. the , i.e. the Codex application server, to Codex application server, to Codex application server, to perform various actions perform various actions perform various actions and receive events. and receive events. But instead of But instead of showing you a bunch of showing you a bunch of showing you a bunch of hypothetical hypothetical hypothetical messages, I went a little messages, I went a little messages, I went a little overboard and overboard and overboard and created a little created a little created a little inspector that inspector that inspector that intercepts all events intercepts all events intercepts all events between the real between the real between the real Codex application and the Codex application and the Codex application and the Codex application server Codex application server so we can so we can so we can take a look at the real take a look at the real take a look at the real events. Let me events. Let me events. Let me switch here switch here switch here to the Codex program. So, to the Codex program. So, to the Codex program. So, I loaded the I loaded the I loaded the same inspector into the same inspector into the same inspector into the built-in browser built-in browser built-in browser here in the Codex app.

here in the Codex app. here in the Codex app. So if we So if we So if we send a send a send a message, we'll message, we'll message, we'll see a bunch of see a bunch of see a bunch of different events different events different events coming in here. coming in here. coming in here. It may seem It may seem It may seem complicated at first, but once you complicated at first, but once you complicated at first, but once you start looking through start looking through start looking through some of these some of these some of these messages, a lot of it messages, a lot of it will seem pretty will seem pretty familiar. So we familiar. So we familiar. So we can review can review can review here...Some of them here...Some of them here...Some of them might be... might be... Let me Let me Let me do that again. I do that again. I think I have some think I have some old old old messages left in my messages left in my messages left in my buffer, so I'll just buffer, so I'll just buffer, so I'll just start a new one. Ahem. That's start a new one. Ahem. That's start a new one. Ahem. That's all. Also, ignore all. Also, ignore my Codex pad up here. But you will see various But you will see various messages, messages, messages, for example, about running for example, about running for example, about running threads with your threads with your threads with your configurations of what is configurations of what is configurations of what is actually actually actually happening, happening, happening, what project we what project we what project we are working on, up to are working on, up to are working on, up to reading the configuration reading the configuration . Um...This is where the . Um...This is where the . Um...This is where the new threads are started new threads are started new threads are started , and here are some of , and here are some of , and here are some of your delta your delta messages, like the messages, like the messages, like the data that's actually data that's actually data that's actually being being being streamed.

streamed. So, you can So, you can find a lot of find a lot of find a lot of additional additional additional information here. And one of the information here. And one of the information here. And one of the main advantages that main advantages that main advantages that makes this protocol makes this protocol makes this protocol so flexible is so flexible is so flexible is that we that we that we open up access open up access open up access to everything in the to everything in the to everything in the application, to all of application, to all of application, to all of its key its key its key functionality. This functionality. This functionality. This means that if we means that if we means that if we stream data or stream data or stream data or perform perform perform file searches, all of this file searches, all of this file searches, all of this information information information is actually transferred between the is actually transferred between the is actually transferred between the application server and the application server and the application server and the application itself. So application itself. So application itself. So if you are building if you are building if you are building your own your own your own harness integration or your own harness integration or your own harness integration or your own client, you can client, you can client, you can use the use the use the same capabilities. same capabilities. In fact, In fact, we currently have over 120 different we currently have over 120 different we currently have over 120 different customer customer customer messages, and we are messages, and we are messages, and we are constantly adding constantly adding constantly adding new ones. This covers everything new ones. This covers everything new ones. This covers everything from initialization, where from initialization, where from initialization, where you can configure you can configure client-specific parameters, to changing client-specific parameters, to changing threads and queues, which are the threads and queues, which are the threads and queues, which are the most most most standard things. standard things. But it also includes But it also includes things like things like things like setting goals setting goals setting goals if you want to if you want to if you want to implement your own implement your own implement your own version of goal mode, version of goal mode, version of goal mode, listing plugins, listing plugins, listing plugins, performing other actions performing other actions performing other actions you see in the you see in the you see in the app, all the way up to searching app, all the way up to searching app, all the way up to searching through files and through files and through files and modifying the file modifying the file modifying the file system. So you system. So you system. So you really get really get really get full access. We full access. We full access. We also try to also try to also try to make it easy for you to make it easy for you to configure the base configure the base agent. And it's a little agent. And it's a little agent. And it's a little different from ACP different from ACP different from ACP because it's a more because it's a more because it's a more general protocol, and general protocol, and general protocol, and we can give you we can give you we can give you more control over more control over more control over individual parts of the individual parts of the individual parts of the harness through this harness through this harness through this protocol. And we'll

protocol. And we'll protocol. And we'll look at that in a moment look at that in a moment . Another thing that . Another thing that . Another thing that excites many is excites many is excites many is that when that when that when using this using this using this protocol, you can protocol, you can protocol, you can add login via add login via add login via ChatGPT. This means that ChatGPT. This means that ChatGPT. This means that your your your app user can app user can app user can use use use their own ChatGPT subscription their own ChatGPT subscription their own ChatGPT subscription and it will not affect and it will not affect Codex usage limits. So if you're Codex usage limits. So if you're using something using something using something like T3 code or something like T3 code or something like T3 code or something like that, you're actually like that, you're actually like that, you're actually using that. using that. Regarding Regarding agent configuration, the two agent configuration, the two agent configuration, the two most important aspects to most important aspects to consider when consider when creating an add-on creating an add-on creating an add-on on top of an application server on top of an application server are are are developer instructions that developer instructions that developer instructions that allow you allow you allow you to supplement the system to supplement the system to supplement the system instructions of the instructions of the instructions of the application itself, as well as application itself, as well as application itself, as well as adding adding adding tools. So, tools. So, tools. So, with the with the developer guidelines, we are developer guidelines, we are actually adding actually adding actually adding data on top of the data on top of the data on top of the Codex system query Codex system query . You can . You can . You can view view view the actual the actual the actual Codex system query in the Codex system query in the Codex system query in the repository. But we repository. But we repository. But we usually usually usually recommend recommend recommend adding to it adding to it adding to it using the using the developer guidelines. This developer guidelines. This way, you continue to way, you continue to way, you continue to use all of use all of use all of our insights our insights our insights into into into query optimization for individual query optimization for individual query optimization for individual models, while models, while models, while adapting them to your adapting them to your adapting them to your specific specific specific harness use case.

harness use case. harness use case. Here's what we Here's what we Here's what we do, for example, when you do, for example, when you do, for example, when you launch a launch a launch a Codex application, Codex application, Codex application, as there are features as there are features as there are features specific to the specific to the specific to the application, as application, as application, as opposed to an opposed to an opposed to an IDE extension or IDE extension or IDE extension or CLI example. Another CLI example. Another CLI example. Another part of this is part of this is dynamic dynamic dynamic tools, where you tools, where you tools, where you can provide can provide can provide additional features additional features additional features that you want the agent to be able to that you want the agent to be able to that you want the agent to be able to use. use. You could You could use MCP or use MCP or use MCP or something similar as an something similar as an agent client protocol, but you could agent client protocol, but you could just define just define just define the tools and the tools and the tools and mark them as mark them as mark them as deferred deferred deferred loading. This loading. This loading. This allows Codex allows Codex allows Codex to use the to use the to use the search tool search tool search tool instead of instead of instead of adding them to the adding them to the adding them to the system query, system query, system query, which we'll talk about which we'll talk about which we'll talk about tomorrow. This allows you tomorrow. This allows you tomorrow. This allows you to find to find to find tools through tools through tools through search without search without search without cluttering the cluttering the cluttering the overall overall overall context window. In addition context window. In addition context window. In addition , you can , you can , you can change any change any change any other settings of other settings of other settings of your agent, even your agent, even your agent, even for each individual for each individual for each individual discussion thread. discussion thread. For example, you can For example, you can change the change the change the model provider model provider , choose a different model, , choose a different model, , choose a different model, or configure or configure or configure the sandbox, including the sandbox, including the sandbox, including restricting access to restricting access to restricting access to network connections or network connections or network connections or files that the agent files that the agent files that the agent can modify. As can modify. As can modify. As for for for using the using the using the application server, the first thing to application server, the first thing to application server, the first thing to know is that it is know is that it is know is that it is part of the Codex CLI. This is the part of the Codex CLI. This is the part of the Codex CLI. This is the same binary file.

same binary file. same binary file. So if you already have the So if you already have the So if you already have the Codex CLI, you can Codex CLI, you can Codex CLI, you can simply simply simply use it use it use it to start the to start the to start the application server right application server right application server right now. You now. You now. You can also see how can also see how can also see how we, for example, we, for example, we, for example, ship it ship it ship it with the with the with the Codex app. The Codex application Codex app. The Codex application Codex app. The Codex application includes this binary includes this binary includes this binary as part of its as part of its as part of its package. If you package. If you package. If you check your check your check your installed installed installed Codex application, you Codex application, you Codex application, you will find it will find it will find it inside the package, and inside the package, and inside the package, and it is updated with it is updated with it is updated with each new version of each new version of each new version of the application. We are the application. We are the application. We are quietly quietly shipping a new shipping a new version of the CLI. On Windows, we version of the CLI. On Windows, we version of the CLI. On Windows, we ship two ship two ship two versions of the file: one versions of the file: one versions of the file: one compiled for Linux compiled for Linux compiled for Linux and one compiled for Windows. This is and one compiled for Windows. This is and one compiled for Windows. This is required so that a Linux file is used in required so that a Linux file is used in required so that a Linux file is used in WSL mode WSL mode , and otherwise a , and otherwise a native native native Windows server. From this Windows server. From this Windows server. From this moment on, you have moment on, you have moment on, you have many many many additional convenient additional convenient additional convenient features. First, you features. First, you features. First, you can ask Codex can ask Codex can ask Codex to generate to generate to generate TypeScript bindings or JSON TypeScript bindings or JSON schema for you for a specific schema for you for a specific schema for you for a specific version of the version of the version of the application server, which is very application server, which is very application server, which is very useful since this useful since this useful since this protocol protocol protocol changes regularly between changes regularly between changes regularly between versions; That's why it's a good idea versions; That's why it's a good idea versions; That's why it's a good idea to add it to add it to add it to your application— to your application— we do this in Codex so that we do this in Codex so that we do this in Codex so that you have full you have full you have full control over the control over the control over the capabilities of the version you're capabilities of the version you're capabilities of the version you're working with. Regarding working with. Regarding working with. Regarding protocols, you can protocols, you can protocols, you can run the run the run the application server via application server via application server via standard input/ standard input/ output (STDIO) or output (STDIO) or output (STDIO) or using websockets using websockets . It depends on . It depends on . It depends on your case, and your case, and your case, and honestly, honestly, honestly, the easiest way the easiest way the easiest way to figure out what to figure out what to figure out what works best for you is to

works best for you is to works best for you is to just ask Codex. just ask Codex. Next are the two Next are the two simplest ways simplest ways simplest ways to integrate. The first to integrate. The first is the Python SDK, which we is the Python SDK, which we is the Python SDK, which we recently released. recently released. recently released. It takes care of the It takes care of the It takes care of the lifecycle management lifecycle management lifecycle management , so you don't , so you don't , so you don't have to have to have to worry about it worry about it worry about it yourself. In addition yourself. In addition , it simplifies the , it simplifies the , it simplifies the integration of features integration of features integration of features such as ChatGPT login such as ChatGPT login such as ChatGPT login and provides a familiar and provides a familiar and provides a familiar interface. However, we are at an interface. However, we are at an interface. However, we are at an AI AI engineering conference, so engineering conference, so engineering conference, so most of you will most of you will most of you will probably choose the probably choose the probably choose the second option— second option— just ask Codex. just ask Codex. just ask Codex. Surprisingly, Codex Surprisingly, Codex implements this implements this protocol extremely well, and you protocol extremely well, and you protocol extremely well, and you mostly mostly mostly just need to point it to the just need to point it to the just need to point it to the documentation. In documentation. In documentation. In most cases, most cases, most cases, this isn't even this isn't even this isn't even necessary, but sometimes necessary, but sometimes necessary, but sometimes it helps to it helps to it helps to give it a push. From here give it a push. From here give it a push. From here you can combine this you can combine this you can combine this with any of our with any of our with any of our plugins, for example plugins, for example plugins, for example to create macOS to create macOS apps if apps if apps if you want to get something up and you want to get something up and you want to get something up and running quickly, or just let your running quickly, or just let your running quickly, or just let your imagination run wild. I, imagination run wild. I, imagination run wild. I, for example, for example, for example, used it used it used it to create to create to create a client in Visual Basic a client in Visual Basic a client in Visual Basic to get a to get a to get a full native full native full native Windows application, and Windows application, and Windows application, and it did it almost on the it did it almost on the it did it almost on the first try. So, first try. So, first try. So, you can build you can build you can build truly impressive truly impressive truly impressive applications in a relatively applications in a relatively applications in a relatively short amount of time without having short amount of time without having short amount of time without having to delve into all the to delve into all the to delve into all the technical details of technical details of technical details of agent development. But the agent development. But the agent development. But the most interesting thing about this most interesting thing about this most interesting thing about this application server application server application server for me is that you are not for me is that you are not for me is that you are not limited to just the limited to just the limited to just the classic classic classic chat chat chat agent interface. You can agent interface. You can agent interface. You can be really be really be really creative: how exactly creative: how exactly creative: how exactly you run this in you run this in you run this in your interface or your interface or your interface or existing app, whether it

existing app, whether it existing app, whether it runs in the runs in the runs in the background, whether background, whether background, whether you install it on you install it on you install it on weird hardware like weird hardware like Maddie does all the time, or if you're Maddie does all the time, or if you're like me, you like me, you like me, you think about how to cram think about how to cram think about how to cram this into a video game. this into a video game. Recently, about a Recently, about a month ago, I month ago, I month ago, I was thinking about how to run was thinking about how to run was thinking about how to run Codex inside Doom, rather than Codex inside Doom, rather than Codex inside Doom, rather than Doom inside Codex, and Doom inside Codex, and Doom inside Codex, and decided to try decided to try decided to try doing it doing it doing it live, so I'll live, so I'll live, so I'll clean up a few clean up a few clean up a few posts now. So, posts now. So, posts now. So, I have the I have the I have the Doom version of the Codex app and I Doom version of the Codex app and I Doom version of the Codex app and I ran it in the wrong ran it in the wrong ran it in the wrong window, let me window, let me window, let me drag it. That's it. drag it. That's it. So, it, uh, So, it, uh, messed up the controls a little bit, messed up the controls a little bit, messed up the controls a little bit, my oversight. One my oversight. One my oversight. One second. Something about this second. Something about this second. Something about this dual- dual- dual- monitor setup sometimes monitor setup sometimes monitor setup sometimes makes it hard to take makes it hard to take makes it hard to take control, so control, so control, so hopefully it'll hopefully it'll hopefully it'll work better now. Yes.

work better now. Yes. work better now. Yes. Okay, this is, uh, an Electron Okay, this is, uh, an Electron app that app that app that uses the Doom Wasm uses the Doom Wasm renderer from Cloudflare. renderer from Cloudflare. Most of the Most of the game game game engine itself is implemented in engine itself is implemented in engine itself is implemented in C, and then the C, and then the FreeDoom game loads, but I had Codex FreeDoom game loads, but I had Codex modify the game file, modify the game file, modify the game file, so as you can see, this so as you can see, this so as you can see, this plays plays plays natively in the video game natively in the video game natively in the video game using their using their using their game engine. And game engine. And game engine. And if we interact with if we interact with if we interact with it here, we it here, we it here, we launch Codex, and launch Codex, and launch Codex, and again, this is again, this is again, this is played played played natively inside the natively inside the natively inside the video game itself, and video game itself, and video game itself, and then it's passed to the then it's passed to the then it's passed to the application server using application server using Electron's IPC protocol. So, if Electron's IPC protocol. So, if we type " we type " hello" here, we'll see hello" here, we'll see hello" here, we'll see that Codex responds. that Codex responds. Version 5.5 is used here. Version 5.5 is used here. The most interesting thing is that you The most interesting thing is that you The most interesting thing is that you can can can use use use dynamic dynamic dynamic tools to tools to tools to add new features. add new features. add new features. Of course, we can Of course, we can Of course, we can get we can get we can get we can get all get all get all the information about armor the information about armor the information about armor and health, the and health, the and health, the tool challenge gives tool challenge gives tool challenge gives me all of that, or me all of that, or me all of that, or even, let's say, all the even, let's say, all the even, let's say, all the weapons with all the ammo weapons with all the ammo . And it calls up different . And it calls up different . And it calls up different tools. So tools. So tools. So it interacts with that, it interacts with that, it interacts with that, but again, but again, but again, remember, it's remember, it's remember, it's still an application server still an application server still an application server running on my running on my running on my computer. So if computer. So if computer. So if I ask, for example, I ask, for example, what is the current directory?

what is the current directory? It actually It actually It actually invokes the Z shell, invokes the Z shell, invokes the Z shell, gets the current gets the current gets the current working directory, and working directory, and working directory, and you see it you see it you see it displays it in displays it in displays it in its own its own its own workspace. So if workspace. So if workspace. So if you want, you can you want, you can you want, you can write code directly in Doom. write code directly in Doom. write code directly in Doom. Not sure I would Not sure I would Not sure I would recommend it. I think the recommend it. I think the recommend it. I think the Codex app is Codex app is Codex app is a bit better suited for this a bit better suited for this . But here you can . But here you can really really really have fun and have fun and have fun and create your own create your own create your own interface. Perfectly. I've interface. Perfectly. I've interface. Perfectly. I've already talked to already talked to already talked to some of you during the some of you during the some of you during the expo about the expo about the expo about the application server, and one application server, and one application server, and one question that came up question that came up question that came up repeatedly is what's the repeatedly is what's the repeatedly is what's the difference with Codex exec. So, difference with Codex exec. So, difference with Codex exec. So, if you've if you've if you've used the used the Codex command line interface before, Codex command line interface before, you might be familiar with this you might be familiar with this you might be familiar with this . There is another . There is another . There is another command called command called command called Codex exec that launches a Codex exec that launches a Codex exec that launches a non-interactive non-interactive non-interactive interface so you interface so you interface so you can submit can submit can submit tasks tasks tasks to Codex for execution. This is to Codex for execution. This is to Codex for execution. This is great for great for great for creating scripts. creating scripts. creating scripts. If you are building a CI/CD If you are building a CI/CD process and want to process and want to process and want to automate certain automate certain automate certain actions, such as actions, such as actions, such as automatically automatically automatically patching PRs, this is a patching PRs, this is a patching PRs, this is a great option. But great option. But great option. But if you are trying to if you are trying to if you are trying to build a more build a more build a more complex client complex client complex client where you interact with an where you interact with an where you interact with an agent, or more agent, or more agent, or more importantly if you importantly if you importantly if you want to want to want to run multiple run multiple run multiple different Codex threads at the same time.

different Codex threads at the same time. For example, you are For example, you are running your own running your own running your own scoring system scoring system scoring system or want to or want to or want to use, use, use, say, slash goal. This is where an say, slash goal. This is where an application server really application server really comes in handy, comes in handy, comes in handy, as you have as you have as you have full control over full control over full control over the environment and don't the environment and don't the environment and don't have to have to have to figure out a way figure out a way figure out a way to parallelize all the to parallelize all the to parallelize all the Codex exec calls yourself Codex exec calls yourself Codex exec calls yourself instead of just instead of just instead of just sending a bunch of sending a bunch of sending a bunch of events to a new thread events to a new thread events to a new thread and running the tasks and running the tasks and running the tasks that way. Before I that way. Before I that way. Before I let you go to the let you go to the let you go to the keynote, keynote, keynote, I have three more tips I have three more tips I have three more tips to wrap up everything that's to wrap up everything that's to wrap up everything that's worth worth worth keeping in mind if you're keeping in mind if you're keeping in mind if you're using an using an using an application server. application server. application server. First, you should First, you should First, you should provide your own provide your own provide your own version of the version of the version of the runtime. This runtime. This runtime. This will save you from will save you from will save you from many problems. many problems. many problems. If you rely If you rely If you rely on the on the on the user-installed CLI, you will user-installed CLI, you will user-installed CLI, you will constantly constantly constantly encounter encounter encounter critical changes, critical changes, critical changes, although we try to although we try to although we try to minimize them, but minimize them, but minimize them, but if you if you if you use use use experimental experimental experimental features, this may features, this may features, this may happen. Plus, you happen. Plus, you happen. Plus, you don't have to don't have to don't have to worry about worry about worry about whether the user has updated whether the user has updated whether the user has updated to to to the latest feature the latest feature the latest feature you want to you want to you want to use, or use, or use, or remind them to remind them to remind them to update. So update. So update. So provide your own provide your own provide your own version of the environment.

version of the environment. version of the environment. The other part is that The other part is that The other part is that you you you should definitely be should definitely be should definitely be opening up new opening up new opening up new features as part of features as part of features as part of your interface. We your interface. We your interface. We do this with the do this with the do this with the Codex app, for example, so Codex app, for example, so Codex app, for example, so it can manage it can manage it can manage itself within the itself within the itself within the Codex flow, triggering Codex flow, triggering Codex flow, triggering automation, etc. automation, etc. automation, etc. But mark them as But mark them as But mark them as deferred. deferred. deferred. This way, you don't This way, you don't unnecessarily clutter the context window with unnecessarily clutter the context window with all those all those all those extra extra extra features you features you features you add, and add, and add, and they remain they remain they remain accessible to the accessible to the accessible to the agent without agent without agent without confusing them. And confusing them. And confusing them. And if you change if you change if you change the instructions, the instructions, the instructions, try to start try to start try to start by adding by adding developer instructions rather than developer instructions rather than changing the base changing the base changing the base instructions. It may instructions. It may instructions. It may seem like you seem like you seem like you want to want to want to write your own write your own write your own system prompt right away, but system prompt right away, but system prompt right away, but there are many different things there are many different things we need to think about constantly when we need to think about constantly when building a system building a system to make sure the to make sure the to make sure the prompt works prompt works prompt works correctly and the agent correctly and the agent correctly and the agent behaves behaves behaves appropriately. And you lose appropriately. And you lose appropriately. And you lose all of that if you're all of that if you're all of that if you're really in a hurry really in a hurry really in a hurry and just write your and just write your and just write your own system own system own system prompt. So be prompt. So be prompt. So be careful if you do careful if you do careful if you do this. That's all, this. That's all, this. That's all, thank you very much. Thank you for thank you very much. Thank you for thank you very much. Thank you for your time.

your time. your time. Scan this QR code Scan this QR code Scan this QR code if you want to get the if you want to get the if you want to get the slides, and I'll be around for slides, and I'll be around for slides, and I'll be around for the next 15 minutes the next 15 minutes the next 15 minutes at the OpenAI booth if at the OpenAI booth if at the OpenAI booth if you want to you want to you want to ask me any ask me any ask me any questions. Thank you very much questions. Thank you very much .