Anthropic Made Claude Even Better

I know most of y'all think of me as that I know most of y'all think of me as that AI dev vibe coding YouTuber, but believe AI dev vibe coding YouTuber, but believe AI dev vibe coding YouTuber, but believe it or not, a few of you were around back it or not, a few of you were around back it or not, a few of you were around back when I was more focused on full stack when I was more focused on full stack when I was more focused on full stack application development, in particular application development, in particular application development, in particular the website, and really focused on the website, and really focused on the website, and really focused on performance and endto-end capabilities performance and endto-end capabilities performance and endto-end capabilities with systems. I love that type of stuff. with systems. I love that type of stuff. with systems. I love that type of stuff. I love nerding out about the details of I love nerding out about the details of I love nerding out about the details of good quality engineering, data loading good quality engineering, data loading good quality engineering, data loading patterns, performance issues, browser patterns, performance issues, browser patterns, performance issues, browser stuff, all those types of things. That's stuff, all those types of things. That's stuff, all those types of things. That's why I'm so excited about this new why I'm so excited about this new why I'm so excited about this new article that Anthropic just put out article that Anthropic just put out article that Anthropic just put out about how they made Cloud.AI AI three about how they made Cloud.AI AI three about how they made Cloud.AI AI three times faster in 2 weeks, largely by times faster in 2 weeks, largely by times faster in 2 weeks, largely by using clot. This is a very fun one for using clot. This is a very fun one for using clot. This is a very fun one for me for a bunch of reasons. Not only am I me for a bunch of reasons. Not only am I me for a bunch of reasons. Not only am I a performance nerd, I've also been a performance nerd, I've also been a performance nerd, I've also been dunking on anthropics performance and dunking on anthropics performance and dunking on anthropics performance and general engineering quality for a long general engineering quality for a long general engineering quality for a long time now. It seems like Anthropics time now. It seems like Anthropics time now. It seems like Anthropics models are now finally good enough that models are now finally good enough that models are now finally good enough that they're actually able to help with they're actually able to help with they're actually able to help with performance instead of hurting it. Which performance instead of hurting it. Which performance instead of hurting it. Which is particularly fun because I posted a is particularly fun because I posted a is particularly fun because I posted a video not that long ago about video not that long ago about video not that long ago about performance issues in T3 code that were performance issues in T3 code that were performance issues in T3 code that were caused by Fable that I had to go solve caused by Fable that I had to go solve caused by Fable that I had to go solve myself. So, I'm very excited to read myself. So, I'm very excited to read myself. So, I'm very excited to read this because one of the things that's this because one of the things that's this because one of the things that's made T3 Code so awesome has been that made T3 Code so awesome has been that made T3 Code so awesome has been that focus on performance. And if Anthropic focus on performance. And if Anthropic focus on performance. And if Anthropic is finally going to stop using 18 gigs is finally going to stop using 18 gigs is finally going to stop using 18 gigs of RAM when I open Cloud Desktop, maybe of RAM when I open Cloud Desktop, maybe of RAM when I open Cloud Desktop, maybe it's more worth a look. I also just want it's more worth a look. I also just want it's more worth a look. I also just want to see how they're setting Cloud up for to see how they're setting Cloud up for to see how they're setting Cloud up for success in these types of environments success in these types of environments success in these types of environments where you're trying to debug deep where you're trying to debug deep where you're trying to debug deep performance issues. It's a not it's a performance issues. It's a not it's a performance issues. It's a not it's a non-trivial thing to do and a lot of the non-trivial thing to do and a lot of the non-trivial thing to do and a lot of the tools that exist, I'll be real, kind of tools that exist, I'll be real, kind of tools that exist, I'll be real, kind of suck. So, I'm very curious how Anthropic suck. So, I'm very curious how Anthropic suck. So, I'm very curious how Anthropic was able to get the profiling set up was able to get the profiling set up was able to get the profiling set up such that they could make the desktop such that they could make the desktop such that they could make the desktop app as well as the website perform app as well as the website perform app as well as the website perform meaningfully better all by using Claude.

meaningfully better all by using Claude. meaningfully better all by using Claude. I do have one more thing worth using I do have one more thing worth using I do have one more thing worth using though, and while it might not be three though, and while it might not be three though, and while it might not be three times faster, it's still worth a listen. times faster, it's still worth a listen. times faster, it's still worth a listen. It's today's sponsor. Weird question for It's today's sponsor. Weird question for It's today's sponsor. Weird question for you. You want people like me to use your you. You want people like me to use your you. You want people like me to use your product? Because if you do, you probably product? Because if you do, you probably product? Because if you do, you probably shouldn't do a normal sign-in flow shouldn't do a normal sign-in flow shouldn't do a normal sign-in flow because I'm not going to be clicking because I'm not going to be clicking because I'm not going to be clicking those buttons myself nowadays. I do most those buttons myself nowadays. I do most those buttons myself nowadays. I do most of my work through my agents now, which of my work through my agents now, which of my work through my agents now, which means any platform that my agents choose means any platform that my agents choose means any platform that my agents choose needs to be usable by the agent. In needs to be usable by the agent. In needs to be usable by the agent. In particular, it needs to be able to be particular, it needs to be able to be particular, it needs to be able to be signed up for by your agents. Today's signed up for by your agents. Today's signed up for by your agents. Today's sponsor is Work OS. And that might sponsor is Work OS. And that might sponsor is Work OS. And that might confuse you because they are a platform confuse you because they are a platform confuse you because they are a platform for user signing in, right? Well, they for user signing in, right? Well, they for user signing in, right? Well, they are. They are the best way to have are. They are the best way to have are. They are the best way to have humans and businesses signing into your humans and businesses signing into your humans and businesses signing into your app. So, if you're trying to get app. So, if you're trying to get app. So, if you're trying to get enterprise customers, then you should enterprise customers, then you should enterprise customers, then you should probably be using work OS. That's why probably be using work OS. That's why probably be using work OS. That's why everyone from OpenAI and Enthropic to everyone from OpenAI and Enthropic to everyone from OpenAI and Enthropic to FA, Cursor, Bolt, Vanta, Carta, and more FA, Cursor, Bolt, Vanta, Carta, and more FA, Cursor, Bolt, Vanta, Carta, and more are all using them. But I want to talk are all using them. But I want to talk are all using them. But I want to talk about OMD because this is a super super about OMD because this is a super super about OMD because this is a super super cool standard. The goal of OMD is to cool standard. The goal of OMD is to cool standard. The goal of OMD is to make it easier for agents to sign up for make it easier for agents to sign up for make it easier for agents to sign up for your apps. It's a new standard that's your apps. It's a new standard that's your apps. It's a new standard that's been designed by works in partnership been designed by works in partnership been designed by works in partnership with companies like Firecrawl and with companies like Firecrawl and with companies like Firecrawl and Cloudflare. And the results speak for Cloudflare. And the results speak for Cloudflare. And the results speak for themselves. There's already a ton of themselves. There's already a ton of themselves. There's already a ton of companies adopting these standards like companies adopting these standards like companies adopting these standards like Neon, Monday, Parallel, and more. What Neon, Monday, Parallel, and more. What Neon, Monday, Parallel, and more. What makes it so cool is that agents can now makes it so cool is that agents can now makes it so cool is that agents can now register on behalf of a user. So if register on behalf of a user. So if register on behalf of a user. So if you're trying to get business customers you're trying to get business customers you're trying to get business customers and you want their agents signing up, and you want their agents signing up, and you want their agents signing up, this is a great place to start. Get off this is a great place to start. Get off this is a great place to start. Get off for all your users at soy.link/workos.

for all your users at soy.link/workos. for all your users at soy.link/workos. Time to dive into how Anthropic made Time to dive into how Anthropic made Time to dive into how Anthropic made Claude three times faster in two weeks. Claude three times faster in two weeks. Claude three times faster in two weeks. My honest guess is that the models are My honest guess is that the models are My honest guess is that the models are much smarter now. So, the quality of much smarter now. So, the quality of much smarter now. So, the quality of Anthropics engineering is actually Anthropics engineering is actually Anthropics engineering is actually improving for the first time in a while. improving for the first time in a while. improving for the first time in a while. But super curious what tooling they set But super curious what tooling they set But super curious what tooling they set up and what they gave Claude access to up and what they gave Claude access to up and what they gave Claude access to in order to make this possible. Once in order to make this possible. Once in order to make this possible. Once Claude can measure something, it can Claude can measure something, it can Claude can measure something, it can make it faster. So, we kept finding more make it faster. So, we kept finding more make it faster. So, we kept finding more things to measure. I like this opening. things to measure. I like this opening. things to measure. I like this opening. Although I am concerned this might be Although I am concerned this might be Although I am concerned this might be the new Claude Pros because they did a the new Claude Pros because they did a the new Claude Pros because they did a lot of work to make Claude less cringe. lot of work to make Claude less cringe. lot of work to make Claude less cringe. But if this becomes the new Claude But if this becomes the new Claude But if this becomes the new Claude format, there will be a new cringe format, there will be a new cringe format, there will be a new cringe sentence structure. They got some nice sentence structure. They got some nice sentence structure. They got some nice little animations on the site, too. It's little animations on the site, too. It's little animations on the site, too. It's cute. But let's dive into the actual cute. But let's dive into the actual cute. But let's dive into the actual details. This August, we made the core details. This August, we made the core details. This August, we made the core user experience of cloud.ai and the user experience of cloud.ai and the user experience of cloud.ai and the Cloud Desktop app around three times Cloud Desktop app around three times Cloud Desktop app around three times faster in a 2 week sprint. Users have faster in a 2 week sprint. Users have faster in a 2 week sprint. Users have been telling us it was slow, and they been telling us it was slow, and they been telling us it was slow, and they were right. Hi, it's me. I'm users. were right. Hi, it's me. I'm users. were right. Hi, it's me. I'm users. Yeah, it it was bad. I did absolutely Yeah, it it was bad. I did absolutely Yeah, it it was bad. I did absolutely notice the difference here when I was notice the difference here when I was notice the difference here when I was going to claw.ai mostly to like check going to claw.ai mostly to like check going to claw.ai mostly to like check the status of my account and my usage the status of my account and my usage the status of my account and my usage before I had my dashboard set up. And I before I had my dashboard set up. And I before I had my dashboard set up. And I noticed the day it got way faster. It noticed the day it got way faster. It noticed the day it got way faster. It was like a meaningful difference. I was like a meaningful difference. I was like a meaningful difference. I didn't think I had a post about it.

didn't think I had a post about it. didn't think I had a post about it. While that's hunting, we could keep While that's hunting, we could keep While that's hunting, we could keep diving in. We focused on four journeys diving in. We focused on four journeys diving in. We focused on four journeys that made up 95% of user activity. At that made up 95% of user activity. At that made up 95% of user activity. At the 75th percentile, time to attage on a the 75th percentile, time to attage on a the 75th percentile, time to attage on a fresh load for Claude went from 3.1 fresh load for Claude went from 3.1 fresh load for Claude went from 3.1 seconds to 0.55. And starting a new seconds to 0.55. And starting a new seconds to 0.55. And starting a new cloud code session went from 0.8 seconds cloud code session went from 0.8 seconds cloud code session went from 0.8 seconds to 0.3. And loading cloud co-work cloud to 0.3. And loading cloud co-work cloud to 0.3. And loading cloud co-work cloud sessions went from almost three seconds sessions went from almost three seconds sessions went from almost three seconds to under one second. In aggregate, we to under one second. In aggregate, we to under one second. In aggregate, we estimate that this saves tens of estimate that this saves tens of estimate that this saves tens of thousands of user hours of waiting every thousands of user hours of waiting every thousands of user hours of waiting every single day. This is why it's important. single day. This is why it's important. single day. This is why it's important. Oh, that's funny. I actually noticed Oh, that's funny. I actually noticed Oh, that's funny. I actually noticed improvements in June. So, they improvements in June. So, they improvements in June. So, they definitely made a bunch of improvements definitely made a bunch of improvements definitely made a bunch of improvements at the time. I even called out that it at the time. I even called out that it at the time. I even called out that it felt faster on a mobile hotspot than it felt faster on a mobile hotspot than it felt faster on a mobile hotspot than it used to feel on gigabit network used to feel on gigabit network used to feel on gigabit network connections. connections. connections. I also found when I dug deeper into it I also found when I dug deeper into it I also found when I dug deeper into it that it was using Tanstack now, which that it was using Tanstack now, which that it was using Tanstack now, which was really cool to see. Specifically, was really cool to see. Specifically, was really cool to see. Specifically, Tanstack router makes it much easier to Tanstack router makes it much easier to Tanstack router makes it much easier to handle these types of navigation cases. handle these types of navigation cases. handle these types of navigation cases. But that makes me even more curious what But that makes me even more curious what But that makes me even more curious what they did because they already did the they did because they already did the they did because they already did the framework upgrade. So, how do they get framework upgrade. So, how do they get framework upgrade. So, how do they get this performance squeezed out after? this performance squeezed out after? this performance squeezed out after? Let's dive into the core user journeys.

Let's dive into the core user journeys. Let's dive into the core user journeys. There's launching the app. They had both There's launching the app. They had both There's launching the app. They had both the web and desktop versions improved the web and desktop versions improved the web and desktop versions improved here. It was, as mentioned before, 3 here. It was, as mentioned before, 3 here. It was, as mentioned before, 3 seconds to launch the website before. seconds to launch the website before. seconds to launch the website before. Now, it's 550 milliseconds. Desktop app Now, it's 550 milliseconds. Desktop app Now, it's 550 milliseconds. Desktop app used to be over 6 seconds and now it's used to be over 6 seconds and now it's used to be over 6 seconds and now it's just over three. Starting conversations, just over three. Starting conversations, just over three. Starting conversations, another thing they focused a lot on and another thing they focused a lot on and another thing they focused a lot on and they were able to make huge improvements they were able to make huge improvements they were able to make huge improvements there. Loading conversations they made there. Loading conversations they made there. Loading conversations they made huge improvements in. And sending a huge improvements in. And sending a huge improvements in. And sending a message, specifically cloud co-work went message, specifically cloud co-work went message, specifically cloud co-work went from a second to 48 milliseconds and cla from a second to 48 milliseconds and cla from a second to 48 milliseconds and cla code on desktop went from 250 code on desktop went from 250 code on desktop went from 250 milliseconds to 52. So this is all in milliseconds to 52. So this is all in milliseconds to 52. So this is all in the apps by the way, not the terminal the apps by the way, not the terminal the apps by the way, not the terminal version, but the cloud code desktop app version, but the cloud code desktop app version, but the cloud code desktop app and the cloud code website. I am and the cloud code website. I am and the cloud code website. I am actually curious how much we feel the actually curious how much we feel the actually curious how much we feel the difference. That's still slow as balls. difference. That's still slow as balls. difference. That's still slow as balls. Still waiting. Still waiting. Still waiting. Still waiting. At this point, I think it Still waiting. At this point, I think it Still waiting. At this point, I think it was a CDN failure. Content at CloudAI was a CDN failure. Content at CloudAI was a CDN failure. Content at CloudAI login may not load or link to file. login may not load or link to file. login may not load or link to file. Great. Refresh. Great. Refresh. Great. Refresh. The refresh worked. I guess those The refresh worked. I guess those The refresh worked. I guess those performance improvements didn't happen performance improvements didn't happen performance improvements didn't happen on the signed out page, but uh now we're on the signed out page, but uh now we're on the signed out page, but uh now we're good. So, let's take a look. I'm going good. So, let's take a look. I'm going good. So, let's take a look. I'm going to do a refresh. That was pretty fast.

to do a refresh. That was pretty fast. to do a refresh. That was pretty fast. Some test prompt. This is one of the Some test prompt. This is one of the Some test prompt. This is one of the ones that pissed me off the most is the ones that pissed me off the most is the ones that pissed me off the most is the flow of submitting a prompt would create flow of submitting a prompt would create flow of submitting a prompt would create the new thread but it wouldn't navigate the new thread but it wouldn't navigate the new thread but it wouldn't navigate which meant that if you refresh there which meant that if you refresh there which meant that if you refresh there was a chance it would disappear forever. was a chance it would disappear forever. was a chance it would disappear forever. That came through very very quickly That came through very very quickly That came through very very quickly actually. Let's do another and it actually. Let's do another and it actually. Let's do another and it immediately refresh. Ah, did they fix immediately refresh. Ah, did they fix immediately refresh. Ah, did they fix that? No, that thread disappeared. Yeah, that? No, that thread disappeared. Yeah, that? No, that thread disappeared. Yeah, that's here's that bug. I hit it again. that's here's that bug. I hit it again. that's here's that bug. I hit it again. This thread that I have open right now This thread that I have open right now This thread that I have open right now did not make it to my recent threads. did not make it to my recent threads. did not make it to my recent threads. So, I have this one from before. This So, I have this one from before. This So, I have this one from before. This one doesn't exist. Make the title proof one doesn't exist. Make the title proof one doesn't exist. Make the title proof of my theory. of my theory. of my theory. I'm going to send this and then refresh I'm going to send this and then refresh I'm going to send this and then refresh really quick. It's hard with one hand. really quick. It's hard with one hand. really quick. It's hard with one hand. Cool. Cool. Cool. Did that said what to make the title getting set up for the session. is getting set up for the session. is really thinking on this. But I did really thinking on this. But I did really thinking on this. But I did manage to kill it getting added to the manage to kill it getting added to the manage to kill it getting added to the sidebar at the very least here.

sidebar at the very least here. sidebar at the very least here. [laughter] It had a failed tool call. [laughter] It had a failed tool call. [laughter] It had a failed tool call. Yeah, this thread effectively doesn't Yeah, this thread effectively doesn't Yeah, this thread effectively doesn't exist because I refreshed before it exist because I refreshed before it exist because I refreshed before it would get persisted to the sidebar, would get persisted to the sidebar, would get persisted to the sidebar, which is uh yeah, rough. It does let you which is uh yeah, rough. It does let you which is uh yeah, rough. It does let you submit the prompt pretty quick though. submit the prompt pretty quick though. submit the prompt pretty quick though. Like if I refresh, Like if I refresh, Like if I refresh, okay, I tried pasting that whole time. okay, I tried pasting that whole time. okay, I tried pasting that whole time. So watch this. I'm going to refresh and So watch this. I'm going to refresh and So watch this. I'm going to refresh and then start spamming paste. It took about then start spamming paste. It took about then start spamming paste. It took about 700 milliseconds or so. A bit under a 700 milliseconds or so. A bit under a 700 milliseconds or so. A bit under a second. Maybe about a second actually. second. Maybe about a second actually. second. Maybe about a second actually. Oh, that one was brutal. That one took Oh, that one was brutal. That one took Oh, that one was brutal. That one took like two to three seconds. While it's like two to three seconds. While it's like two to three seconds. While it's still not meeting my bar, I am impressed still not meeting my bar, I am impressed still not meeting my bar, I am impressed overall with the difference. It does overall with the difference. It does overall with the difference. It does feel a lot nicer to navigate. I hope feel a lot nicer to navigate. I hope feel a lot nicer to navigate. I hope they can figure out Oh, look there. My they can figure out Oh, look there. My they can figure out Oh, look there. My threads finally appeared. Took them a threads finally appeared. Took them a threads finally appeared. Took them a bit, but they did. But yeah, you get my bit, but they did. But yeah, you get my bit, but they did. But yeah, you get my point. It has a lot of like data point. It has a lot of like data point. It has a lot of like data persistence layer issues still, but it persistence layer issues still, but it persistence layer issues still, but it does feel better. I see people in chat does feel better. I see people in chat does feel better. I see people in chat saying they need convex. Unironically, saying they need convex. Unironically, saying they need convex. Unironically, yes, it helps with so many of these yes, it helps with so many of these yes, it helps with so many of these things. I don't know how well it would things. I don't know how well it would things. I don't know how well it would handle the insane load that I'm sure handle the insane load that I'm sure handle the insane load that I'm sure that Anthropic pushes, but something that Anthropic pushes, but something that Anthropic pushes, but something similar to it with a proper engine that similar to it with a proper engine that similar to it with a proper engine that keeps stuff up to date would help a lot.

keeps stuff up to date would help a lot. keeps stuff up to date would help a lot. A few months ago, I did a deep dive on A few months ago, I did a deep dive on A few months ago, I did a deep dive on how both ChatGpt and Cloud.AI I were how both ChatGpt and Cloud.AI I were how both ChatGpt and Cloud.AI I were handling caching for the data that was handling caching for the data that was handling caching for the data that was being fetched on the page that you being fetched on the page that you being fetched on the page that you actually saw and learned that both are actually saw and learned that both are actually saw and learned that both are now caching the recent threads in the now caching the recent threads in the now caching the recent threads in the sidebar through local storage or I guess sidebar through local storage or I guess sidebar through local storage or I guess now in this case now I'm digging into it now in this case now I'm digging into it now in this case now I'm digging into it index DB is a way of using a key value index DB is a way of using a key value index DB is a way of using a key value store to store the threads you have in store to store the threads you have in store to store the threads you have in the sidebar. All these threads here are the sidebar. All these threads here are the sidebar. All these threads here are all cached when you load which is why all cached when you load which is why all cached when you load which is why when I refresh they come in almost when I refresh they come in almost when I refresh they come in almost immediately and there's no pop in or immediately and there's no pop in or immediately and there's no pop in or flash in. To show you a really silly flash in. To show you a really silly flash in. To show you a really silly example of like why this matters, I'm example of like why this matters, I'm example of like why this matters, I'm going to go to my other browser and going to go to my other browser and going to go to my other browser and archive a handful of these. Uh, I guess archive a handful of these. Uh, I guess archive a handful of these. Uh, I guess there's no real archive, so I just have there's no real archive, so I just have there's no real archive, so I just have to delete. That's fine. Delete. Delete. to delete. That's fine. Delete. Delete. to delete. That's fine. Delete. Delete. Cool. So, I deleted those two. So, Cool. So, I deleted those two. So, Cool. So, I deleted those two. So, there's still one test prompt. Watch there's still one test prompt. Watch there's still one test prompt. Watch what happens when I refresh. They're not what happens when I refresh. They're not what happens when I refresh. They're not invalidating the cache. It still thinks invalidating the cache. It still thinks invalidating the cache. It still thinks these two threads exist when they don't. these two threads exist when they don't. these two threads exist when they don't. How do I get this to refresh? If I click How do I get this to refresh? If I click How do I get this to refresh? If I click that doesn't even do it. What? I was that doesn't even do it. What? I was that doesn't even do it. What? I was hoping this was going to be a video hoping this was going to be a video hoping this was going to be a video about how much Anthropics improve their about how much Anthropics improve their about how much Anthropics improve their engineering quality. I'm almost relieved engineering quality. I'm almost relieved engineering quality. I'm almost relieved to see nothing has changed. What the to see nothing has changed. What the to see nothing has changed. What the I I have these two threads that I I have these two threads that I I have these two threads that when I click, you get an error, but it when I click, you get an error, but it when I click, you get an error, but it never revalidates the sidebar. You know never revalidates the sidebar. You know never revalidates the sidebar. You know what? I bet I bet if I send a prompt what? I bet I bet if I send a prompt what? I bet I bet if I send a prompt here that will Nope, that updates the here that will Nope, that updates the here that will Nope, that updates the status for that one. But these two are status for that one. But these two are status for that one. But these two are still broken. Chat just said coding is still broken. Chat just said coding is still broken. Chat just said coding is solved, by the way. Yep. Yep. Coding solved, by the way. Yep. Yep. Coding solved, by the way. Yep. Yep. Coding solved. AGI is here.

solved. AGI is here. solved. AGI is here. [groaning] [groaning] [groaning] They dis No, they didn't disappear. They dis No, they didn't disappear. They dis No, they didn't disappear. They're still here. Two broken They're still here. Two broken They're still here. Two broken threads. Are you kidding? I I hope threads. Are you kidding? I I hope threads. Are you kidding? I I hope you'll understand why it's a convex you'll understand why it's a convex you'll understand why it's a convex shell now. It solves all of these types shell now. It solves all of these types shell now. It solves all of these types of problems. of problems. of problems. Even T3 code doesn't do this. We don't Even T3 code doesn't do this. We don't Even T3 code doesn't do this. We don't even have a server to sync against. It's even have a server to sync against. It's even have a server to sync against. It's all on your devices. Oh, they disappear. all on your devices. Oh, they disappear. all on your devices. Oh, they disappear. When I was complaining, they just poof When I was complaining, they just poof When I was complaining, they just poof vanished all of a sudden. I was even vanished all of a sudden. I was even vanished all of a sudden. I was even looking. God. Reminder that according to looking. God. Reminder that according to looking. God. Reminder that according to Sam Alman, I'm an anthropic shell. So Sam Alman, I'm an anthropic shell. So Sam Alman, I'm an anthropic shell. So yeah, take it as you will. >> [sighs] >> [sighs] >> Well, that answers a bunch of my >> Well, that answers a bunch of my >> Well, that answers a bunch of my questions. They made performance better questions. They made performance better questions. They made performance better by making it so that things never get by making it so that things never get by making it so that things never get updated. Clever. They're caching far too updated. Clever. They're caching far too updated. Clever. They're caching far too aggressively, and they're not aggressively, and they're not aggressively, and they're not invalidating the cache anywhere near invalidating the cache anywhere near invalidating the cache anywhere near often enough. For reference, in T3 code, often enough. For reference, in T3 code, often enough. For reference, in T3 code, we do this a bit differently when we're we do this a bit differently when we're we do this a bit differently when we're in the loading state because we haven't in the loading state because we haven't in the loading state because we haven't fetched the new data from the backend fetched the new data from the backend fetched the new data from the backend yet. I show it in a faded blurry state yet. I show it in a faded blurry state yet. I show it in a faded blurry state because I don't want to show you because I don't want to show you because I don't want to show you confidence in data that isn't accurate confidence in data that isn't accurate confidence in data that isn't accurate yet. The fade and doesn't trigger until yet. The fade and doesn't trigger until yet. The fade and doesn't trigger until we have loaded the real data from the we have loaded the real data from the we have loaded the real data from the server. Did I say T3 code? I'm sorry. T3 server. Did I say T3 code? I'm sorry. T3 server. Did I say T3 code? I'm sorry. T3 chat. My bad. You guys know what I'm chat. My bad. You guys know what I'm chat. My bad. You guys know what I'm talking about. T3 code does everything talking about. T3 code does everything talking about. T3 code does everything through servers that you're running through servers that you're running through servers that you're running yourself. So, it's very different here.

yourself. So, it's very different here. yourself. So, it's very different here. The caching like caching just works The caching like caching just works The caching like caching just works fundamentally different when there's one fundamentally different when there's one fundamentally different when there's one database per user and it's on your own database per user and it's on your own database per user and it's on your own machine. For T3 chat, you need to do machine. For T3 chat, you need to do machine. For T3 chat, you need to do more. And as incredible as Convex is, more. And as incredible as Convex is, more. And as incredible as Convex is, and it absolutely is. I'm so happy we and it absolutely is. I'm so happy we and it absolutely is. I'm so happy we moved our data over there. It still moved our data over there. It still moved our data over there. It still doesn't give you data as soon as the doesn't give you data as soon as the doesn't give you data as soon as the page loads. I want data as soon as the page loads. I want data as soon as the page loads. I want data as soon as the page loads, which is why we set this up page loads, which is why we set this up page loads, which is why we set this up with the local storage cache so that we with the local storage cache so that we with the local storage cache so that we can show that immediately. I did also can show that immediately. I did also can show that immediately. I did also notice though that we changed how we are notice though that we changed how we are notice though that we changed how we are loading in things on the bottom bar. We loading in things on the bottom bar. We loading in things on the bottom bar. We need to start caching that as well. On need to start caching that as well. On need to start caching that as well. On that note, actually, let me um new that note, actually, let me um new that note, actually, let me um new thread T3 chat. I notice that it seems thread T3 chat. I notice that it seems thread T3 chat. I notice that it seems like we're not caching the necessary like we're not caching the necessary like we're not caching the necessary data in order to open the model picker data in order to open the model picker data in order to open the model picker and other things at the bottom of the and other things at the bottom of the and other things at the bottom of the composer as soon as the page loads. It's composer as soon as the page loads. It's composer as soon as the page loads. It's kind of jarring to have the sidebar come kind of jarring to have the sidebar come kind of jarring to have the sidebar come in immediately, but the model picker and in immediately, but the model picker and in immediately, but the model picker and other selectors in the chat take some other selectors in the chat take some other selectors in the chat take some time to load. What would it look like to time to load. What would it look like to time to load. What would it look like to cache that similar to how we cache on cache that similar to how we cache on cache that similar to how we cache on the sidebar? If it's relatively easy, I the sidebar? If it's relatively easy, I the sidebar? If it's relatively easy, I would like you to implement it and file would like you to implement it and file would like you to implement it and file a PR and babyset it. As much as I like a PR and babyset it. As much as I like a PR and babyset it. As much as I like the dunk on anthropics lack of the dunk on anthropics lack of the dunk on anthropics lack of engineering quality, I think that we engineering quality, I think that we engineering quality, I think that we should go through this article because should go through this article because should go through this article because at the very least they did make a lot of at the very least they did make a lot of at the very least they did make a lot of improvements to performance. And while improvements to performance. And while improvements to performance. And while they seem to have regressed a bunch of they seem to have regressed a bunch of they seem to have regressed a bunch of data patterns in the process, we can data patterns in the process, we can data patterns in the process, we can still learn from the ways that they gave still learn from the ways that they gave still learn from the ways that they gave Claude access to the things it needs in Claude access to the things it needs in Claude access to the things it needs in order to find these performance issues order to find these performance issues order to find these performance issues and fix them. Let's start reading though and fix them. Let's start reading though and fix them. Let's start reading though cuz I am actually excited. Apparently, cuz I am actually excited. Apparently, cuz I am actually excited. Apparently, they did the whole thing through Claude they did the whole thing through Claude they did the whole thing through Claude tag. So, they didn't even use cloud code tag. So, they didn't even use cloud code tag. So, they didn't even use cloud code or I don't know something better like T3 or I don't know something better like T3 or I don't know something better like T3 code. They did this all through Claude code. They did this all through Claude code. They did this all through Claude Tag, which is the Slackbot. I have seen Tag, which is the Slackbot. I have seen Tag, which is the Slackbot. I have seen examples from Boris, the guy who started

examples from Boris, the guy who started examples from Boris, the guy who started Cloud Code, using Claude Tag to do all Cloud Code, using Claude Tag to do all Cloud Code, using Claude Tag to do all of his work every day. He actually of his work every day. He actually of his work every day. He actually showed an example of a real thread he showed an example of a real thread he showed an example of a real thread he did to add the mods feature to uh codeex did to add the mods feature to uh codeex did to add the mods feature to uh codeex or to cloud code if I recall. I could be or to cloud code if I recall. I could be or to cloud code if I recall. I could be wrong on that, but he did something like wrong on that, but he did something like wrong on that, but he did something like that recently. So, they used Claude Tag that recently. So, they used Claude Tag that recently. So, they used Claude Tag using an internal research model roughly using an internal research model roughly using an internal research model roughly comparable to Opus 55. Huh. I hope this comparable to Opus 55. Huh. I hope this comparable to Opus 55. Huh. I hope this is just a snapshot of what became 55 and is just a snapshot of what became 55 and is just a snapshot of what became 55 and not some other model that they're not some other model that they're not some other model that they're claiming is similar to 55. Like they claiming is similar to 55. Like they claiming is similar to 55. Like they claimed Opus 5 was similar to Fable when claimed Opus 5 was similar to Fable when claimed Opus 5 was similar to Fable when it wasn't. But uh yeah, no guardrails it wasn't. But uh yeah, no guardrails it wasn't. But uh yeah, no guardrails though, which is a good call out from though, which is a good call out from though, which is a good call out from Maria because when they're testing the Maria because when they're testing the Maria because when they're testing the models, they have much less guardrails models, they have much less guardrails models, they have much less guardrails generally. And I happen to know that generally. And I happen to know that generally. And I happen to know that when you start diving into Chrome when you start diving into Chrome when you start diving into Chrome internals, it becomes much more likely internals, it becomes much more likely internals, it becomes much more likely that you start to hit the safety guards that you start to hit the safety guards that you start to hit the safety guards and the stuff that prevents you from and the stuff that prevents you from and the stuff that prevents you from sending prompts because it thinks you sending prompts because it thinks you sending prompts because it thinks you might be hacking. So yeah, I will put might be hacking. So yeah, I will put might be hacking. So yeah, I will put that all out there because I think it's that all out there because I think it's that all out there because I think it's important because I've had that important because I've had that important because I've had that experience. I am hopeful because again I experience. I am hopeful because again I experience. I am hopeful because again I haven't really pushed Opus too hard haven't really pushed Opus too hard haven't really pushed Opus too hard here. Actually, I'm kind of lying there. here. Actually, I'm kind of lying there. here. Actually, I'm kind of lying there. I did have one instance. Let me find it I did have one instance. Let me find it I did have one instance. Let me find it in my terminal quick. I vaguely remember in my terminal quick. I vaguely remember in my terminal quick. I vaguely remember you mentioning that the other agent hit you mentioning that the other agent hit you mentioning that the other agent hit some type of security flag or guard rail some type of security flag or guard rail some type of security flag or guard rail when it was working on the performance when it was working on the performance when it was working on the performance improvements. Do you have more info on improvements. Do you have more info on improvements. Do you have more info on that? Like what it was trying to do at that? Like what it was trying to do at that? Like what it was trying to do at the time and what guardrail it hit? It the time and what guardrail it hit? It the time and what guardrail it hit? It was a built-in safety check on remove was a built-in safety check on remove was a built-in safety check on remove commands that I hit. So, not as bad as I commands that I hit. So, not as bad as I commands that I hit. So, not as bad as I thought. I assumed I was hitting the thought. I assumed I was hitting the thought. I assumed I was hitting the guard rails because of the weird things guard rails because of the weird things guard rails because of the weird things that Opus was doing in Chrome to debug that Opus was doing in Chrome to debug that Opus was doing in Chrome to debug performance stuff for fish slop. It does performance stuff for fish slop. It does performance stuff for fish slop. It does appear to be the case that that isn't appear to be the case that that isn't appear to be the case that that isn't what happened here. So hopefully we

what happened here. So hopefully we what happened here. So hopefully we won't hit too many of those guardrails. won't hit too many of those guardrails. won't hit too many of those guardrails. I've seen it before with the other I've seen it before with the other I've seen it before with the other models. I haven't done as much deep models. I haven't done as much deep models. I haven't done as much deep diving in performance stuff with clawed diving in performance stuff with clawed diving in performance stuff with clawed models lately. Honestly, I have still models lately. Honestly, I have still models lately. Honestly, I have still found OpenAI models to be better at this found OpenAI models to be better at this found OpenAI models to be better at this type of uh I still call it the like type of uh I still call it the like type of uh I still call it the like Rottweiler way of working where it grabs Rottweiler way of working where it grabs Rottweiler way of working where it grabs the problem by the neck and strangles the problem by the neck and strangles the problem by the neck and strangles it. I have found anthropic models to not it. I have found anthropic models to not it. I have found anthropic models to not be quite as good as OpenAI models in the be quite as good as OpenAI models in the be quite as good as OpenAI models in the GPT line for these like deep dive tear GPT line for these like deep dive tear GPT line for these like deep dive tear apart all the performance problems type apart all the performance problems type apart all the performance problems type stuff. But a lot of this does just come stuff. But a lot of this does just come stuff. But a lot of this does just come down to the tooling because codeex has down to the tooling because codeex has down to the tooling because codeex has much better computer use stuff built in much better computer use stuff built in much better computer use stuff built in and it seems like it's able to figure and it seems like it's able to figure and it seems like it's able to figure out how to control Chrome and get these out how to control Chrome and get these out how to control Chrome and get these traces out of it a little bit better. traces out of it a little bit better. traces out of it a little bit better. Bullets still aren't great at it from my Bullets still aren't great at it from my Bullets still aren't great at it from my experience though and I find myself experience though and I find myself experience though and I find myself still digging in to the details myself still digging in to the details myself still digging in to the details myself when I'm deep in performance stuff. So when I'm deep in performance stuff. So when I'm deep in performance stuff. So this is all why I'm so curious how this is all why I'm so curious how this is all why I'm so curious how Anthropic did this. They said that Anthropic did this. They said that Anthropic did this. They said that through cloud tag, they were able to through cloud tag, they were able to through cloud tag, they were able to find bottlenecks, build benchmarks, ship find bottlenecks, build benchmarks, ship find bottlenecks, build benchmarks, ship improvements, and watch every deploy. improvements, and watch every deploy. improvements, and watch every deploy. They steered by setting goals, making They steered by setting goals, making They steered by setting goals, making trade-offs, and improving every change. trade-offs, and improving every change. trade-offs, and improving every change. With that approach, they merged more With that approach, they merged more With that approach, they merged more than 3,000 changes without a single than 3,000 changes without a single than 3,000 changes without a single customerf facing incident or roll back. customerf facing incident or roll back. customerf facing incident or roll back. To be fair, I did just kind of showcase To be fair, I did just kind of showcase To be fair, I did just kind of showcase things that might have been worthwhile things that might have been worthwhile things that might have been worthwhile uh a roll back or an incident, but uh to uh a roll back or an incident, but uh to uh a roll back or an incident, but uh to each the own. This is a post that covers each the own. This is a post that covers each the own. This is a post that covers how they did it apparently safely. Very how they did it apparently safely. Very how they did it apparently safely. Very interesting. Before the sprint, they interesting. Before the sprint, they interesting. Before the sprint, they created a new channel in Slack with the created a new channel in Slack with the created a new channel in Slack with the following standing instructions. At following standing instructions. At following standing instructions. At Claude, your job is to facilitate all Claude, your job is to facilitate all Claude, your job is to facilitate all things related to the performance of the things related to the performance of the things related to the performance of the Claude AI website and desktop app. Your Claude AI website and desktop app. Your Claude AI website and desktop app. Your responsibilities include monitoring responsibilities include monitoring responsibilities include monitoring deploys for performance regressions, deploys for performance regressions, deploys for performance regressions, assessing the accuracy and assessing the accuracy and assessing the accuracy and comprehensiveness of existing telemetry, comprehensiveness of existing telemetry, comprehensiveness of existing telemetry, maintaining well-curated observability

maintaining well-curated observability maintaining well-curated observability dashboards, proactively implementing dashboards, proactively implementing dashboards, proactively implementing solutions for observed issues and solutions for observed issues and solutions for observed issues and low-hanging fruit, proposing performance low-hanging fruit, proposing performance low-hanging fruit, proposing performance project opportunities, and communicating project opportunities, and communicating project opportunities, and communicating with your human teammates. The ultimate with your human teammates. The ultimate with your human teammates. The ultimate goal for this channel is for you to goal for this channel is for you to goal for this channel is for you to become as autonomous as possible, but become as autonomous as possible, but become as autonomous as possible, but today we know that isn't yet possible. today we know that isn't yet possible. today we know that isn't yet possible. This is a terrible prompt. They didn't This is a terrible prompt. They didn't This is a terrible prompt. They didn't even mention to make no mistakes. No even mention to make no mistakes. No even mention to make no mistakes. No wonder it had those regressions. If they wonder it had those regressions. If they wonder it had those regressions. If they had just told it make no mistakes, why had just told it make no mistakes, why had just told it make no mistakes, why would it had no problems? We asked would it had no problems? We asked would it had no problems? We asked Claude to analyze usage data through the Claude to analyze usage data through the Claude to analyze usage data through the data dog MCP server. They got that dog data dog MCP server. They got that dog data dog MCP server. They got that dog in them. It identified the four highest in them. It identified the four highest in them. It identified the four highest impact user journeys. The app launches, impact user journeys. The app launches, impact user journeys. The app launches, starting conversations, loading existing starting conversations, loading existing starting conversations, loading existing conversations, and sending messages conversations, and sending messages conversations, and sending messages between web and desktop, as well as between web and desktop, as well as between web and desktop, as well as across our products. Those journeys came across our products. Those journeys came across our products. Those journeys came to 13 distinct measurements. To to 13 distinct measurements. To to 13 distinct measurements. To establish our baselines, we added establish our baselines, we added establish our baselines, we added instrumentation until they were directly instrumentation until they were directly instrumentation until they were directly comparable. Each started with a user comparable. Each started with a user comparable. Each started with a user interaction, ended once the result was interaction, ended once the result was interaction, ended once the result was rendered, and disamiguated client and rendered, and disamiguated client and rendered, and disamiguated client and server work. People underrate how server work. People underrate how server work. People underrate how important this part is to fixing important this part is to fixing important this part is to fixing performance issues. It is much harder to performance issues. It is much harder to performance issues. It is much harder to fix what you can't measure, even if fix what you can't measure, even if fix what you can't measure, even if you're not an AI agent. I cannot tell you're not an AI agent. I cannot tell you're not an AI agent. I cannot tell you how many arguments I would get into you how many arguments I would get into you how many arguments I would get into in my previous jobs where I was like, in my previous jobs where I was like, in my previous jobs where I was like, "This thing is slow as hell." They're "This thing is slow as hell." They're "This thing is slow as hell." They're like, "Well, we looked in our dashboard.

like, "Well, we looked in our dashboard. like, "Well, we looked in our dashboard. and only took 200 milliseconds. And then and only took 200 milliseconds. And then and only took 200 milliseconds. And then I went and profiled it myself and I went and profiled it myself and I went and profiled it myself and learned that they were counting from not learned that they were counting from not learned that they were counting from not like when the page was loaded to when like when the page was loaded to when like when the page was loaded to when the thing happened. They were counting the thing happened. They were counting the thing happened. They were counting from like when their component rendered from like when their component rendered from like when their component rendered in the UI to when it went from loading in the UI to when it went from loading in the UI to when it went from loading state to done state. And if it took 8 state to done state. And if it took 8 state to done state. And if it took 8 seconds to get there, then your trace seconds to get there, then your trace seconds to get there, then your trace doesn't matter. And so many places get doesn't matter. And so many places get doesn't matter. And so many places get their profiling wrong with all of these their profiling wrong with all of these their profiling wrong with all of these types of things. I cannot even types of things. I cannot even types of things. I cannot even comprehend how many times I have been comprehend how many times I have been comprehend how many times I have been shown data to prove me wrong about a shown data to prove me wrong about a shown data to prove me wrong about a performance issue just to learn that the performance issue just to learn that the performance issue just to learn that the data was not actually reflecting the data was not actually reflecting the data was not actually reflecting the real world performance at all. Now let's real world performance at all. Now let's real world performance at all. Now let's see how they actually got the work done see how they actually got the work done see how they actually got the work done once they did the instrumentation. We once they did the instrumentation. We once they did the instrumentation. We kicked off the sprint with a list of kicked off the sprint with a list of kicked off the sprint with a list of about 20 handpicked projects each about 20 handpicked projects each about 20 handpicked projects each targeting specific journeys. Claude targeting specific journeys. Claude targeting specific journeys. Claude estimated the impact of each project in estimated the impact of each project in estimated the impact of each project in milliseconds and we aggregated those milliseconds and we aggregated those milliseconds and we aggregated those estimates to set our targets for the estimates to set our targets for the estimates to set our targets for the sprint. Some of the projects were fairly sprint. Some of the projects were fairly sprint. Some of the projects were fairly large but we thought we could probably large but we thought we could probably large but we thought we could probably achieve most of them within 2 weeks. achieve most of them within 2 weeks. achieve most of them within 2 weeks. They hit 12 of their 13 targets by day They hit 12 of their 13 targets by day They hit 12 of their 13 targets by day three. Did they fall for the classic three. Did they fall for the classic three. Did they fall for the classic mistake? I was planning on having Opus mistake? I was planning on having Opus mistake? I was planning on having Opus do the classic rewrite of ping.gg to do the classic rewrite of ping.gg to do the classic rewrite of ping.gg to make it modern. I'm curious about a make it modern. I'm curious about a make it modern. I'm curious about a thing quick though. How long do you thing quick though. How long do you thing quick though. How long do you think these changes would take if we did think these changes would take if we did think these changes would take if we did them all in one pass? Here we are. The them all in one pass? Here we are. The them all in one pass? Here we are. The the classic anthropic overestimation. As the classic anthropic overestimation. As the classic anthropic overestimation. As great as models are now at writing code, great as models are now at writing code, great as models are now at writing code, understanding code, their understanding understanding code, their understanding understanding code, their understanding of agents is still kind of garbage. And of agents is still kind of garbage. And of agents is still kind of garbage. And as such, when you ask them how long will as such, when you ask them how long will as such, when you ask them how long will it take to do something, they're wrong.

it take to do something, they're wrong. it take to do something, they're wrong. This is an overhaul that I'm pretty sure This is an overhaul that I'm pretty sure This is an overhaul that I'm pretty sure it could do in a few hours, and it is it could do in a few hours, and it is it could do in a few hours, and it is very confident it would add up to about very confident it would add up to about very confident it would add up to about 25 working days to do it. It is just 25 working days to do it. It is just 25 working days to do it. It is just wrong, but it's cute that it thinks it wrong, but it's cute that it thinks it wrong, but it's cute that it thinks it isn't wrong. I'm going to throw this one isn't wrong. I'm going to throw this one isn't wrong. I'm going to throw this one a a rough instruction here. I want you a a rough instruction here. I want you a a rough instruction here. I want you to kick off a agent run on my computer to kick off a agent run on my computer to kick off a agent run on my computer left book. You should be able to see how left book. You should be able to see how left book. You should be able to see how to connect through the fleet repo on to connect through the fleet repo on to connect through the fleet repo on this machine. I want you to make sure this machine. I want you to make sure this machine. I want you to make sure this repo is cloned. All the environment this repo is cloned. All the environment this repo is cloned. All the environment variables that are needed are there and variables that are needed are there and variables that are needed are there and that you can run opus 5.5 through cloud that you can run opus 5.5 through cloud that you can run opus 5.5 through cloud code in T3 code such that I can still code in T3 code such that I can still code in T3 code such that I can still see the thread how I normally do in T3 see the thread how I normally do in T3 see the thread how I normally do in T3 code. My goal here is to do the whole code. My goal here is to do the whole code. My goal here is to do the whole implementation on that machine in one implementation on that machine in one implementation on that machine in one pass with however many sub aents and pass with however many sub aents and pass with however many sub aents and whatever else are needed with full whatever else are needed with full whatever else are needed with full computer use capability on the other computer use capability on the other computer use capability on the other machine because I don't want you to be machine because I don't want you to be machine because I don't want you to be spamming my machine with computer use spamming my machine with computer use spamming my machine with computer use requests and browser control while I'm requests and browser control while I'm requests and browser control while I'm using it. If anything does not work as using it. If anything does not work as using it. If anything does not work as expected when you move this workload expected when you move this workload expected when you move this workload over to Leftbook, don't hesitate to ask over to Leftbook, don't hesitate to ask over to Leftbook, don't hesitate to ask questions so we can get it working questions so we can get it working questions so we can get it working right. I'm actually curious how that right. I'm actually curious how that right. I'm actually curious how that work goes. We'll check into that in a work goes. We'll check into that in a work goes. We'll check into that in a bit. But yeah, point I was trying to bit. But yeah, point I was trying to bit. But yeah, point I was trying to make by pulling this up is to highlight make by pulling this up is to highlight make by pulling this up is to highlight how bad models are at estimating time how bad models are at estimating time how bad models are at estimating time frames cuz they don't understand how frames cuz they don't understand how frames cuz they don't understand how capable models are, which I honestly capable models are, which I honestly capable models are, which I honestly find hilarious. I'm just trying to do an find hilarious. I'm just trying to do an find hilarious. I'm just trying to do an extended version of the joke that Claude extended version of the joke that Claude extended version of the joke that Claude told them it would take 2 weeks and then told them it would take 2 weeks and then told them it would take 2 weeks and then it actually took 3 days cuz these guys it actually took 3 days cuz these guys it actually took 3 days cuz these guys make Claude makes sense. They're bad at make Claude makes sense. They're bad at make Claude makes sense. They're bad at judging this too, as we were. The judging this too, as we were. The judging this too, as we were. The planned projects landed early. For planned projects landed early. For planned projects landed early. For faster launches, we baked a static faster launches, we baked a static faster launches, we baked a static composer into the HTML so users can type

composer into the HTML so users can type composer into the HTML so users can type during the React initialization. That's during the React initialization. That's during the React initialization. That's a fun one that we did a while back. They a fun one that we did a while back. They a fun one that we did a while back. They also pre-ompiled the V8 code cache so also pre-ompiled the V8 code cache so also pre-ompiled the V8 code cache so the desktop shell main process doesn't the desktop shell main process doesn't the desktop shell main process doesn't have to recompile it from scratch. Oh have to recompile it from scratch. Oh have to recompile it from scratch. Oh boy, they're going deep in the weeds on boy, they're going deep in the weeds on boy, they're going deep in the weeds on that one. H, do I want to add that? You that one. H, do I want to add that? You that one. H, do I want to add that? You know what? know what? know what? Anthropic has been focused on Anthropic has been focused on Anthropic has been focused on overhauling the performance for cloud.I overhauling the performance for cloud.I overhauling the performance for cloud.I and the cloud code desktop app. I and the cloud code desktop app. I and the cloud code desktop app. I noticed this paragraph in particular has noticed this paragraph in particular has noticed this paragraph in particular has some fun ideas on how they improve some fun ideas on how they improve some fun ideas on how they improve performance. I'm curious if any of these performance. I'm curious if any of these performance. I'm curious if any of these would make sense for us. both the would make sense for us. both the would make sense for us. both the embedding the HTML so the composer is embedding the HTML so the composer is embedding the HTML so the composer is available faster but more importantly available faster but more importantly available faster but more importantly the V8 cacheed compile I'm curious if the V8 cacheed compile I'm curious if the V8 cacheed compile I'm curious if we'd be able to implement something like we'd be able to implement something like we'd be able to implement something like that in T3 code and what type of that in T3 code and what type of that in T3 code and what type of performance improvements we would see if performance improvements we would see if performance improvements we would see if we did very curious how that goes we did very curious how that goes we did very curious how that goes another fun Reacty change is that in another fun Reacty change is that in another fun Reacty change is that in order to make navigation faster they order to make navigation faster they order to make navigation faster they kept the composer mounted between kept the composer mounted between kept the composer mounted between conversations and they also prefetched conversations and they also prefetched conversations and they also prefetched sessions when the user hovered over them sessions when the user hovered over them sessions when the user hovered over them and in doing all of that also cut cyber and in doing all of that also cut cyber and in doing all of that also cut cyber rrenders by 90%. These are the types of rrenders by 90%. These are the types of rrenders by 90%. These are the types of things I was doing by hand just barely a things I was doing by hand just barely a things I was doing by hand just barely a year ago in order to squeeze every bit year ago in order to squeeze every bit year ago in order to squeeze every bit of performance I could out of T3 chat.

of performance I could out of T3 chat. of performance I could out of T3 chat. Just some fun real examples here. I'm Just some fun real examples here. I'm Just some fun real examples here. I'm just going to click a random thread so just going to click a random thread so just going to click a random thread so you can see how fast it loads. That was you can see how fast it loads. That was you can see how fast it loads. That was not as good as I was hoping. If I go to not as good as I was hoping. If I go to not as good as I was hoping. If I go to a different one, skill manager name. a different one, skill manager name. a different one, skill manager name. Okay, that was immediate. That's Okay, that was immediate. That's Okay, that was immediate. That's immediate. That's immediate. I want immediate. That's immediate. I want immediate. That's immediate. I want independent status. Immediate. Yeah, independent status. Immediate. Yeah, independent status. Immediate. Yeah, data fetching on Hover is one of the data fetching on Hover is one of the data fetching on Hover is one of the most fun little tricks you can do to most fun little tricks you can do to most fun little tricks you can do to make things feel absurdly fast. It's make things feel absurdly fast. It's make things feel absurdly fast. It's actually crazy how fast things will feel actually crazy how fast things will feel actually crazy how fast things will feel when you make a small change like that. when you make a small change like that. when you make a small change like that. I am surprised they hadn't done it I am surprised they hadn't done it I am surprised they hadn't done it before. They also called out that they before. They also called out that they before. They also called out that they left room for Claude to identify left room for Claude to identify left room for Claude to identify opportunities and propose new work opportunities and propose new work opportunities and propose new work streams. Those work streams quickly streams. Those work streams quickly streams. Those work streams quickly ramped into full projects of their own, ramped into full projects of their own, ramped into full projects of their own, which far exceeded their initial which far exceeded their initial which far exceeded their initial targets. So, they set new targets and targets. So, they set new targets and targets. So, they set new targets and looked for more things to measure after. looked for more things to measure after. looked for more things to measure after. We've ended up funding nearly every We've ended up funding nearly every We've ended up funding nearly every project in the original project list and project in the original project list and project in the original project list and more funding in completing. Curious the more funding in completing. Curious the more funding in completing. Curious the wording there. Let's do a refresh. What wording there. Let's do a refresh. What wording there. Let's do a refresh. What have we not explored and what can we have we not explored and what can we have we not explored and what can we hill climb on? Where's the most hill climb on? Where's the most hill climb on? Where's the most opportunity at this point? I am open to opportunity at this point? I am open to opportunity at this point? I am open to wacky ideas. Don't underestimate the wacky ideas. Don't underestimate the wacky ideas. Don't underestimate the value of that particular piece of the value of that particular piece of the value of that particular piece of the prompt. I know it sounds crazy. I'll be prompt. I know it sounds crazy. I'll be prompt. I know it sounds crazy. I'll be real. It's because it kind of is crazy.

real. It's because it kind of is crazy. real. It's because it kind of is crazy. When I was testing GBD6 Astra early, I When I was testing GBD6 Astra early, I When I was testing GBD6 Astra early, I tried something similar here. I was tried something similar here. I was tried something similar here. I was doing a huge performance overhaul for doing a huge performance overhaul for doing a huge performance overhaul for Lakebed, my cloud, trying to really see Lakebed, my cloud, trying to really see Lakebed, my cloud, trying to really see how far I could push performance on it. how far I could push performance on it. how far I could push performance on it. So, these small number of user tiny So, these small number of user tiny So, these small number of user tiny JavaScript apps in the cloud would use JavaScript apps in the cloud would use JavaScript apps in the cloud would use as little resource as possible. It did a as little resource as possible. It did a as little resource as possible. It did a bunch of runs and the performance was bunch of runs and the performance was bunch of runs and the performance was better after the first performance set better after the first performance set better after the first performance set of improvements that I merged, but it of improvements that I merged, but it of improvements that I merged, but it wasn't quite where I wanted. So, I wrote wasn't quite where I wanted. So, I wrote wasn't quite where I wanted. So, I wrote a prompt very similar to the one we just a prompt very similar to the one we just a prompt very similar to the one we just read. Sounds like there's more read. Sounds like there's more read. Sounds like there's more improvements to be made. I want you to improvements to be made. I want you to improvements to be made. I want you to really dig deep and find everything we really dig deep and find everything we really dig deep and find everything we can, even extreme measures to improve can, even extreme measures to improve can, even extreme measures to improve performance. I want you to come to me performance. I want you to come to me performance. I want you to come to me with three different proposals for with three different proposals for with three different proposals for things we could do. The more extreme the things we could do. The more extreme the things we could do. The more extreme the better. Don't be afraid to boil the better. Don't be afraid to boil the better. Don't be afraid to boil the ocean. And this is why I now have a ocean. And this is why I now have a ocean. And this is why I now have a custom runtime for LakeBed. I ended up custom runtime for LakeBed. I ended up custom runtime for LakeBed. I ended up forking Rusty V8 and building my own forking Rusty V8 and building my own forking Rusty V8 and building my own runtime in order to make the performance runtime in order to make the performance runtime in order to make the performance as good as possible. And it was like a 2 as good as possible. And it was like a 2 as good as possible. And it was like a 2 to 4x improvement. They proposed the to 4x improvement. They proposed the to 4x improvement. They proposed the persistent V8 execution service which persistent V8 execution service which persistent V8 execution service which ended up being its own custom runtime. ended up being its own custom runtime. ended up being its own custom runtime. Build a query compiler and maintain live Build a query compiler and maintain live Build a query compiler and maintain live answers and give each app a durable answers and give each app a durable answers and give each app a durable owner with local data. didn't end up owner with local data. didn't end up owner with local data. didn't end up doing those other two because I just got doing those other two because I just got doing those other two because I just got so focused in on the persistent V8 so focused in on the persistent V8 so focused in on the persistent V8 execution service. Rusty V8 is a execution service. Rusty V8 is a execution service. Rusty V8 is a phenomenal library by the way. Thank you phenomenal library by the way. Thank you phenomenal library by the way. Thank you to the maintainers of that. But yeah, I to the maintainers of that. But yeah, I to the maintainers of that. But yeah, I effectively told it to just go hill effectively told it to just go hill effectively told it to just go hill climb, come up with something crazy and climb, come up with something crazy and climb, come up with something crazy and the classic boil the ocean and it did the classic boil the ocean and it did the classic boil the ocean and it did it. It made something awesome and the it. It made something awesome and the it. It made something awesome and the performance is crazy. And for those performance is crazy. And for those performance is crazy. And for those asking where's Lake Bed, it's become my asking where's Lake Bed, it's become my asking where's Lake Bed, it's become my pet project to play with these things pet project to play with these things pet project to play with these things with. So, uh, making it like a real with. So, uh, making it like a real with. So, uh, making it like a real product that I can confidently pitch to product that I can confidently pitch to product that I can confidently pitch to y'all is less a focus for me now. But it y'all is less a focus for me now. But it y'all is less a focus for me now. But it is out there. It works. If you find the is out there. It works. If you find the is out there. It works. If you find the URL, which I don't hide very well in the

URL, which I don't hide very well in the URL, which I don't hide very well in the npm package, which is pretty easy to npm package, which is pretty easy to npm package, which is pretty easy to find, too. You can probably use it for find, too. You can probably use it for find, too. You can probably use it for real stuff. I use it all the time. Hell, real stuff. I use it all the time. Hell, real stuff. I use it all the time. Hell, I used it yesterday in my Opus 55 video I used it yesterday in my Opus 55 video I used it yesterday in my Opus 55 video to make a better visualizer for the data to make a better visualizer for the data to make a better visualizer for the data on the uh, artificial analysis dashes. on the uh, artificial analysis dashes. on the uh, artificial analysis dashes. So much easier to read this than it is So much easier to read this than it is So much easier to read this than it is to read the artificial analysis to read the artificial analysis to read the artificial analysis homepage. It's a fun little thing. It's homepage. It's a fun little thing. It's homepage. It's a fun little thing. It's very nice to just like set up apps that very nice to just like set up apps that very nice to just like set up apps that autosync and live update and get stuff autosync and live update and get stuff autosync and live update and get stuff out. But the performance improvements I out. But the performance improvements I out. But the performance improvements I got out of this really stupid prompt and got out of this really stupid prompt and got out of this really stupid prompt and also a lot of time spent working with it also a lot of time spent working with it also a lot of time spent working with it to build both ways to verify the to build both ways to verify the to build both ways to verify the performance improvements as well as ways performance improvements as well as ways performance improvements as well as ways to verify the behavior not regressing. to verify the behavior not regressing. to verify the behavior not regressing. It was a a war. This is one of the It was a a war. This is one of the It was a a war. This is one of the longest threads I've ever done in T3 longest threads I've ever done in T3 longest threads I've ever done in T3 code. And a lot of pretty crazy run code. And a lot of pretty crazy run code. And a lot of pretty crazy run times, too. Like the planning took 20 times, too. Like the planning took 20 times, too. Like the planning took 20 minutes. The first build was about 2 minutes. The first build was about 2 minutes. The first build was about 2 hours and then something broke. I think hours and then something broke. I think hours and then something broke. I think I was doing something on the machine so I was doing something on the machine so I was doing something on the machine so it stopped. I told to continue. Five and it stopped. I told to continue. Five and it stopped. I told to continue. Five and a half hours later it had the PR and it a half hours later it had the PR and it a half hours later it had the PR and it actually did the thing and it ran actually did the thing and it ran actually did the thing and it ran awesome. So yeah, don't underestimate awesome. So yeah, don't underestimate awesome. So yeah, don't underestimate something as silly as I am open to wacky something as silly as I am open to wacky something as silly as I am open to wacky ideas in a prompt like this. And that's ideas in a prompt like this. And that's ideas in a prompt like this. And that's why the next quote here sounds insane, why the next quote here sounds insane, why the next quote here sounds insane, but I I'm on their side here. Anything but I I'm on their side here. Anything but I I'm on their side here. Anything can be hill climbed. They're not wrong.

can be hill climbed. They're not wrong. can be hill climbed. They're not wrong. From the start, we knew we wanted to From the start, we knew we wanted to From the start, we knew we wanted to iterate faster than our deploy cadence. iterate faster than our deploy cadence. iterate faster than our deploy cadence. Claude could work asynchronously for Claude could work asynchronously for Claude could work asynchronously for many hours, even overnight. We wanted to many hours, even overnight. We wanted to many hours, even overnight. We wanted to let it validate its prototypes without let it validate its prototypes without let it validate its prototypes without waiting for field reads. To achieve waiting for field reads. To achieve waiting for field reads. To achieve that, we looked for other ways to that, we looked for other ways to that, we looked for other ways to measure performance in the lab. Sam measure performance in the lab. Sam measure performance in the lab. Sam found the first lead. What can we do found the first lead. What can we do found the first lead. What can we do instead of wall clock timing? Can we instead of wall clock timing? Can we instead of wall clock timing? Can we measure JS instruction counts for measure JS instruction counts for measure JS instruction counts for instance? Yeah, for pure JS hot paths, instance? Yeah, for pure JS hot paths, instance? Yeah, for pure JS hot paths, literal instruction counts, run the literal instruction counts, run the literal instruction counts, run the benchmark under valindrind with node benchmark under valindrind with node benchmark under valindrind with node predictable and compare to a checkedin predictable and compare to a checkedin predictable and compare to a checkedin baseline. One run, no statistics needed. baseline. One run, no statistics needed. baseline. One run, no statistics needed. Okay, so we know this isn't Opus 55. Okay, so we know this isn't Opus 55. Okay, so we know this isn't Opus 55. There's one little hint that this wasn't There's one little hint that this wasn't There's one little hint that this wasn't Opus 55. Thank you for including such a Opus 55. Thank you for including such a Opus 55. Thank you for including such a useful hint in topic for browser paths. useful hint in topic for browser paths. useful hint in topic for browser paths. There's no instruction counting under There's no instruction counting under There's no instruction counting under Chromium, but there's a ladder of other Chromium, but there's a ladder of other Chromium, but there's a ladder of other deterministic counts. React commits per deterministic counts. React commits per deterministic counts. React commits per interaction, function call counts from interaction, function call counts from interaction, function call counts from V8's precise coverage, layout and style V8's precise coverage, layout and style V8's precise coverage, layout and style recount counts, DOM mutations. What do recount counts, DOM mutations. What do recount counts, DOM mutations. What do you want first? He said, "Let's explore you want first? He said, "Let's explore you want first? He said, "Let's explore val grind plus IR and predictable flags val grind plus IR and predictable flags val grind plus IR and predictable flags in one thread in each of the browser in one thread in each of the browser in one thread in each of the browser React benches in new threads. Ping me in React benches in new threads. Ping me in React benches in new threads. Ping me in all of them. You know what we want.

all of them. You know what we want. all of them. You know what we want. Let's go." I I I hate to be so fixated Let's go." I I I hate to be so fixated Let's go." I I I hate to be so fixated on this. I have not read too many on this. I have not read too many on this. I have not read too many prompts from employees of the labs. I prompts from employees of the labs. I prompts from employees of the labs. I don't know how I feel that my prompting don't know how I feel that my prompting don't know how I feel that my prompting is so similar to the prompting of is so similar to the prompting of is so similar to the prompting of anthropic employees. I am a combination anthropic employees. I am a combination anthropic employees. I am a combination of proud and ashamed in this moment. of proud and ashamed in this moment. of proud and ashamed in this moment. Audio made fun of me for the way I Audio made fun of me for the way I Audio made fun of me for the way I prompt. I feel better about it now. prompt. I feel better about it now. prompt. I feel better about it now. Anyways, Anyways, Anyways, 11 minutes later, five threads were 11 minutes later, five threads were 11 minutes later, five threads were running, each focused on different running, each focused on different running, each focused on different measurements. Why it take 11 minutes to measurements. Why it take 11 minutes to measurements. Why it take 11 minutes to spin up five threads? Regardless, spin up five threads? Regardless, spin up five threads? Regardless, instruction counts, V8 call counts, instruction counts, V8 call counts, instruction counts, V8 call counts, React commits, style recalc, and DOM React commits, style recalc, and DOM React commits, style recalc, and DOM manipulations. I think I know what manipulations. I think I know what manipulations. I think I know what happened now to those weird issues I was happened now to those weird issues I was happened now to those weird issues I was calling out before. If you're just calling out before. If you're just calling out before. If you're just blindly measuring the number of React blindly measuring the number of React blindly measuring the number of React commits or V8 calls, you're going to commits or V8 calls, you're going to commits or V8 calls, you're going to start turning things off that don't start turning things off that don't start turning things off that don't actually affect performance. The harsh actually affect performance. The harsh actually affect performance. The harsh reality is that when you're on the page reality is that when you're on the page reality is that when you're on the page and the page is working, firing more and the page is working, firing more and the page is working, firing more network requests is not too big a deal, network requests is not too big a deal, network requests is not too big a deal, especially on like desktop web and especially on like desktop web and especially on like desktop web and desktop app experiences. I think both desktop app experiences. I think both desktop app experiences. I think both people and agents are a little too quick people and agents are a little too quick people and agents are a little too quick to look at every single network request to look at every single network request to look at every single network request and try to disable them rather than find and try to disable them rather than find and try to disable them rather than find ways to change the hierarchy of requests ways to change the hierarchy of requests ways to change the hierarchy of requests to get you what you need to interact to get you what you need to interact to get you what you need to interact with the page first and then keep things with the page first and then keep things with the page first and then keep things up to date and reliable after. And the up to date and reliable after. And the up to date and reliable after. And the reason I'm calling that out now is reason I'm calling that out now is reason I'm calling that out now is giving the agent this type of insight is giving the agent this type of insight is giving the agent this type of insight is encouraging that type of behavior encouraging that type of behavior encouraging that type of behavior because you're not going to see when the because you're not going to see when the because you're not going to see when the page becomes interactive via V8 call page becomes interactive via V8 call page becomes interactive via V8 call counts, but you are going to see every counts, but you are going to see every counts, but you are going to see every request happening whether or not it's request happening whether or not it's request happening whether or not it's blocking. And if you see opportunities

blocking. And if you see opportunities blocking. And if you see opportunities to reduce those requests, like I don't to reduce those requests, like I don't to reduce those requests, like I don't know, maybe we shouldn't fetch all of know, maybe we shouldn't fetch all of know, maybe we shouldn't fetch all of the sidebar data all the time. We the sidebar data all the time. We the sidebar data all the time. We already have it in local storage. That already have it in local storage. That already have it in local storage. That would kill six of the 800 calls on this would kill six of the 800 calls on this would kill six of the 800 calls on this page in this trace. we should turn that page in this trace. we should turn that page in this trace. we should turn that off. And then you end up in the state off. And then you end up in the state off. And then you end up in the state that I showed earlier where the app that I showed earlier where the app that I showed earlier where the app wasn't syncing the sidebar content wasn't syncing the sidebar content wasn't syncing the sidebar content properly and you could end up with bad properly and you could end up with bad properly and you could end up with bad sidebar runs. Not the best solution sidebar runs. Not the best solution sidebar runs. Not the best solution here. It is useful tooling, but you need here. It is useful tooling, but you need here. It is useful tooling, but you need to also make sure the agent is combining to also make sure the agent is combining to also make sure the agent is combining that with real introspection to how the that with real introspection to how the that with real introspection to how the user flow works. I don't necessarily user flow works. I don't necessarily user flow works. I don't necessarily know if agents can do the whole end to know if agents can do the whole end to know if agents can do the whole end to end here yet. That's why I still do end here yet. That's why I still do end here yet. That's why I still do performance stuff very much in the loop performance stuff very much in the loop performance stuff very much in the loop myself. I'll usually start by asking it myself. I'll usually start by asking it myself. I'll usually start by asking it to come up with theories to improve to come up with theories to improve to come up with theories to improve performance for things that could be performance for things that could be performance for things that could be causing a specific thing to be slow. causing a specific thing to be slow. causing a specific thing to be slow. Like I'll start with it takes too long Like I'll start with it takes too long Like I'll start with it takes too long to start typing in the input box. What to start typing in the input box. What to start typing in the input box. What do you think we can do to improve it? do you think we can do to improve it? do you think we can do to improve it? Then I'll tell it to go make demos with Then I'll tell it to go make demos with Then I'll tell it to go make demos with all of those potential improvements and all of those potential improvements and all of those potential improvements and then I will test it myself to see if it then I will test it myself to see if it then I will test it myself to see if it feels faster or not. It is far too easy feels faster or not. It is far too easy feels faster or not. It is far too easy to cause weird niche UX regressions in to cause weird niche UX regressions in to cause weird niche UX regressions in performance overhauls. And I still do performance overhauls. And I still do performance overhauls. And I still do this a lot myself. They then asked it to this a lot myself. They then asked it to this a lot myself. They then asked it to prove that hill climbing against each of prove that hill climbing against each of prove that hill climbing against each of these can actually result in measurable these can actually result in measurable these can actually result in measurable wall clock performance wins. We'll wall clock performance wins. We'll wall clock performance wins. We'll unship the benches for any candidates unship the benches for any candidates unship the benches for any candidates that cannot prove that. Interesting.

that cannot prove that. Interesting. that cannot prove that. Interesting. They also called that they treated every They also called that they treated every They also called that they treated every new benchmark with skepticism. Each one new benchmark with skepticism. Each one new benchmark with skepticism. Each one had two jobs. First, make a metric that had two jobs. First, make a metric that had two jobs. First, make a metric that Claude could move in the lab. And Claude could move in the lab. And Claude could move in the lab. And second, a guardrail in CI with a number second, a guardrail in CI with a number second, a guardrail in CI with a number that could only ratchet down. This is that could only ratchet down. This is that could only ratchet down. This is one of my favorites. Benchmark was flaky one of my favorites. Benchmark was flaky one of my favorites. Benchmark was flaky or didn't actually correlate with user or didn't actually correlate with user or didn't actually correlate with user latency. We threw it out rather than latency. We threw it out rather than latency. We threw it out rather than letting Claude climb the wrong hill. letting Claude climb the wrong hill. letting Claude climb the wrong hill. Also very good they called that out Also very good they called that out Also very good they called that out there. I hate that I'm just going to there. I hate that I'm just going to there. I hate that I'm just going to like keep deep diving into my own stuff, like keep deep diving into my own stuff, like keep deep diving into my own stuff, but that's what happens when we talk but that's what happens when we talk but that's what happens when we talk about performance. I get to nerd out about performance. I get to nerd out about performance. I get to nerd out about my favorite I somewhat about my favorite I somewhat about my favorite I somewhat recently did a lot of work in T3 Code's recently did a lot of work in T3 Code's recently did a lot of work in T3 Code's data layer to make it so you don't have data layer to make it so you don't have data layer to make it so you don't have to load as much data when you open big to load as much data when you open big to load as much data when you open big threads or you resume the app. That's a threads or you resume the app. That's a threads or you resume the app. That's a big part of why T3 Code feels so fast big part of why T3 Code feels so fast big part of why T3 Code feels so fast even on absolutely horrifying hell even on absolutely horrifying hell even on absolutely horrifying hell threads. For example, this build Swift threads. For example, this build Swift threads. For example, this build Swift UI app from scratch thread. This thread UI app from scratch thread. This thread UI app from scratch thread. This thread I've been working on for 4 months and it I've been working on for 4 months and it I've been working on for 4 months and it probably has like 500 prompts in it at probably has like 500 prompts in it at probably has like 500 prompts in it at this point. Watch how fast it loads when this point. Watch how fast it loads when this point. Watch how fast it loads when I click. I am clicking now. Pretty much I click. I am clicking now. Pretty much I click. I am clicking now. Pretty much immediate. It did have like a flicker immediate. It did have like a flicker immediate. It did have like a flicker down due to a weird scroll state thing. down due to a weird scroll state thing. down due to a weird scroll state thing. Fun problem for me to fix later, but Fun problem for me to fix later, but Fun problem for me to fix later, but things load pretty much immediately in things load pretty much immediately in things load pretty much immediately in T3 code because I had to go the hell and T3 code because I had to go the hell and T3 code because I had to go the hell and back finding every single place I could back finding every single place I could back finding every single place I could squeeze out data that wasn't necessary squeeze out data that wasn't necessary squeeze out data that wasn't necessary in the loads. I made a lot of these in the loads. I made a lot of these in the loads. I made a lot of these improvements and then noticed things improvements and then noticed things improvements and then noticed things starting to get worse again. And I starting to get worse again. And I starting to get worse again. And I didn't want that. And I also wanted to didn't want that. And I also wanted to didn't want that. And I also wanted to be easier to measure when I made be easier to measure when I made be easier to measure when I made improvements. Which is why the most improvements. Which is why the most improvements. Which is why the most important change I made actually had important change I made actually had important change I made actually had nothing to do with the performance nothing to do with the performance nothing to do with the performance improvements itself. The thing that improvements itself. The thing that improvements itself. The thing that mattered is this guy. I added an action mattered is this guy. I added an action mattered is this guy. I added an action that would do a test for a bunch of

that would do a test for a bunch of that would do a test for a bunch of different request formats that actually different request formats that actually different request formats that actually exist in the app with some stored data exist in the app with some stored data exist in the app with some stored data for a real in quotes fake thread to make for a real in quotes fake thread to make for a real in quotes fake thread to make sure that we are not meaningfully going sure that we are not meaningfully going sure that we are not meaningfully going above or below the existing data like above or below the existing data like above or below the existing data like baseline that we have for loading stuff baseline that we have for loading stuff baseline that we have for loading stuff and this has prevented so many and this has prevented so many and this has prevented so many regressions. I am very thankful I added regressions. I am very thankful I added regressions. I am very thankful I added this because not only will the GitHub this because not only will the GitHub this because not only will the GitHub action catch these types of regressions action catch these types of regressions action catch these types of regressions by putting it in as a comment, both the by putting it in as a comment, both the by putting it in as a comment, both the agent working on the code as well as the agent working on the code as well as the agent working on the code as well as the agents doing PR review will notice it agents doing PR review will notice it agents doing PR review will notice it and call it out and potentially even fix and call it out and potentially even fix and call it out and potentially even fix it autonomously. So this gives a signal it autonomously. So this gives a signal it autonomously. So this gives a signal to the agents when they make mistakes to the agents when they make mistakes to the agents when they make mistakes that cause performance regressions so that cause performance regressions so that cause performance regressions so they can fix it before it becomes my they can fix it before it becomes my they can fix it before it becomes my problem or worse my user's problem. And problem or worse my user's problem. And problem or worse my user's problem. And once again, I want to emphasize, even if once again, I want to emphasize, even if once again, I want to emphasize, even if you're not that into using vibe coding you're not that into using vibe coding you're not that into using vibe coding into AI coded stuff to ship realworld into AI coded stuff to ship realworld into AI coded stuff to ship realworld projects to users, if you're not using projects to users, if you're not using projects to users, if you're not using it for some of your work, you are it for some of your work, you are it for some of your work, you are missing out so much because using these missing out so much because using these missing out so much because using these tools to slop together tests like this tools to slop together tests like this tools to slop together tests like this where like, yeah, if this was where like, yeah, if this was where like, yeah, if this was missionritical to have our performance missionritical to have our performance missionritical to have our performance never regress, it might be worth really never regress, it might be worth really never regress, it might be worth really digging into these tests to make sure digging into these tests to make sure digging into these tests to make sure every detail of them is good. But that every detail of them is good. But that every detail of them is good. But that was not the mindset I went in for this.

was not the mindset I went in for this. was not the mindset I went in for this. It was a much more vague, wouldn't it be It was a much more vague, wouldn't it be It was a much more vague, wouldn't it be cool if we had a better idea when cool if we had a better idea when cool if we had a better idea when regressions kind of happened. And it regressions kind of happened. And it regressions kind of happened. And it threw together this test suite. I think threw together this test suite. I think threw together this test suite. I think it's a couple thousand lines. I don't it's a couple thousand lines. I don't it's a couple thousand lines. I don't know. I didn't read the code. know. I didn't read the code. know. I didn't read the code. And now it has actively prevented And now it has actively prevented And now it has actively prevented regressions in our codebase. So once regressions in our codebase. So once regressions in our codebase. So once again, slop is useful even if you never again, slop is useful even if you never again, slop is useful even if you never ship it to your users. Back to the ship it to your users. Back to the ship it to your users. Back to the article. Wall clock time is what users article. Wall clock time is what users article. Wall clock time is what users feel, but it's noisy and milliseconds feel, but it's noisy and milliseconds feel, but it's noisy and milliseconds are too flaky to use as CI gates. are too flaky to use as CI gates. are too flaky to use as CI gates. Instruction counts were appealing Instruction counts were appealing Instruction counts were appealing because they were deterministic, but we because they were deterministic, but we because they were deterministic, but we still needed Claude to prove they still needed Claude to prove they still needed Claude to prove they tracked wall clock time. We asked Claude tracked wall clock time. We asked Claude tracked wall clock time. We asked Claude to drive the countdown on two hot paths. to drive the countdown on two hot paths. to drive the countdown on two hot paths. The routine that assembles a The routine that assembles a The routine that assembles a conversation's message tree and a conversation's message tree and a conversation's message tree and a scanner for status lines in Claude code scanner for status lines in Claude code scanner for status lines in Claude code output. Claude profiled both with output. Claude profiled both with output. Claude profiled both with Valgrren and found that a quarter of its Valgrren and found that a quarter of its Valgrren and found that a quarter of its first paths instructions were first paths instructions were first paths instructions were megamorphic dictionary lookups resolving megamorphic dictionary lookups resolving megamorphic dictionary lookups resolving the same message ID three separate the same message ID three separate the same message ID three separate times. Going to be so real with y'all on times. Going to be so real with y'all on times. Going to be so real with y'all on this one. Data loading patterns tend to this one. Data loading patterns tend to this one. Data loading patterns tend to be the vast majority of the problems be the vast majority of the problems be the vast majority of the problems with these types of things. Like if it with these types of things. Like if it with these types of things. Like if it is going down the same path three times, is going down the same path three times, is going down the same path three times, that means data is being fetched in a that means data is being fetched in a that means data is being fetched in a way that triggers rerenders three times way that triggers rerenders three times way that triggers rerenders three times usually. An hour later, it cut usually. An hour later, it cut usually. An hour later, it cut instructions on both paths by 48% and instructions on both paths by 48% and instructions on both paths by 48% and 31%. And while clock time had dropped 31%. And while clock time had dropped 31%. And while clock time had dropped from 78% and 44%. We checked in two new from 78% and 44%. We checked in two new from 78% and 44%. We checked in two new ratchets. From then on, an APR that ratchets. From then on, an APR that ratchets. From then on, an APR that raised the instruction counts of these raised the instruction counts of these raised the instruction counts of these paths would fail CI. the daily job paths would fail CI. the daily job paths would fail CI. the daily job lowered each ceiling whenever the count lowered each ceiling whenever the count lowered each ceiling whenever the count went down. That's another good idea. If went down. That's another good idea. If went down. That's another good idea. If you do make improvements, it's important you do make improvements, it's important you do make improvements, it's important to update these things because if you to update these things because if you to update these things because if you don't, then you end up in a weird place don't, then you end up in a weird place don't, then you end up in a weird place where every performance improvement

where every performance improvement where every performance improvement becomes area for somebody to make things becomes area for somebody to make things becomes area for somebody to make things worse. Again, this is even more painful worse. Again, this is even more painful worse. Again, this is even more painful with unit testing and code coverage. I with unit testing and code coverage. I with unit testing and code coverage. I hate code coverage, period. Like the hate code coverage, period. Like the hate code coverage, period. Like the idea of like having a number that shows idea of like having a number that shows idea of like having a number that shows you how well tested your code is is you how well tested your code is is you how well tested your code is is silly and dumb. And I had times at silly and dumb. And I had times at silly and dumb. And I had times at Twitch where I had a feature that I Twitch where I had a feature that I Twitch where I had a feature that I rewrote. We took like half a million rewrote. We took like half a million rewrote. We took like half a million lines of code and rewrote it in 50k and lines of code and rewrote it in 50k and lines of code and rewrote it in 50k and we had shipped the new version with a we had shipped the new version with a we had shipped the new version with a feature flag to trigger between the two. feature flag to trigger between the two. feature flag to trigger between the two. We had moved everyone over to the new We had moved everyone over to the new We had moved everyone over to the new version and it was great. And then I version and it was great. And then I version and it was great. And then I went to delete the other 500,000 lines went to delete the other 500,000 lines went to delete the other 500,000 lines of code and couldn't because when I did of code and couldn't because when I did of code and couldn't because when I did that, the code bases code coverage would that, the code bases code coverage would that, the code bases code coverage would go down from 95% to 94. It was not like go down from 95% to 94. It was not like go down from 95% to 94. It was not like it was like 94.9 or whatever, but it was it was like 94.9 or whatever, but it was it was like 94.9 or whatever, but it was enough that it made it so I couldn't enough that it made it so I couldn't enough that it made it so I couldn't merge the PR. Well, you should add more merge the PR. Well, you should add more merge the PR. Well, you should add more tests to your changes then. I did. My tests to your changes then. I did. My tests to your changes then. I did. My 50k lines had 100% coverage. The 500k 50k lines had 100% coverage. The 500k 50k lines had 100% coverage. The 500k lines had 96%. But that was enough where lines had 96%. But that was enough where lines had 96%. But that was enough where deleting it would shift the code base deleting it would shift the code base deleting it would shift the code base enough to knock us just below the enough to knock us just below the enough to knock us just below the threshold. And I never got to delete threshold. And I never got to delete threshold. And I never got to delete this giant old unused feature that we this giant old unused feature that we this giant old unused feature that we had rewritten because of code coverage.

had rewritten because of code coverage. had rewritten because of code coverage. Does the count track the clock? CPU Does the count track the clock? CPU Does the count track the clock? CPU instructions versus wall clock time. instructions versus wall clock time. instructions versus wall clock time. Yeah, CPU instruction was 48% less. wall Yeah, CPU instruction was 48% less. wall Yeah, CPU instruction was 48% less. wall clock was 78% less for message tree clock was 78% less for message tree clock was 78% less for message tree assembly. That's really the best way to assembly. That's really the best way to assembly. That's really the best way to measure that. Also doesn't give us real measure that. Also doesn't give us real measure that. Also doesn't give us real times for wall clock here. So we don't times for wall clock here. So we don't times for wall clock here. So we don't know if it matters cuz like if you shave know if it matters cuz like if you shave know if it matters cuz like if you shave 200 milliseconds to 60 milliseconds that 200 milliseconds to 60 milliseconds that 200 milliseconds to 60 milliseconds that matters a lot less than 4 seconds down matters a lot less than 4 seconds down matters a lot less than 4 seconds down to half a second. This led them to the to half a second. This led them to the to half a second. This led them to the central lesson of the sprint with central lesson of the sprint with central lesson of the sprint with Claude. Measuring something makes it Claude. Measuring something makes it Claude. Measuring something makes it tractable. Yeah. Makes it so you can tractable. Yeah. Makes it so you can tractable. Yeah. Makes it so you can actually give Claude the things it needs actually give Claude the things it needs actually give Claude the things it needs to improve once it can measure. to improve once it can measure. to improve once it can measure. Measurement used to be step zero. You'd Measurement used to be step zero. You'd Measurement used to be step zero. You'd add a metric, wait for data to roll in, add a metric, wait for data to roll in, add a metric, wait for data to roll in, and only then would you start to and only then would you start to and only then would you start to understand the problem. With Claude, understand the problem. With Claude, understand the problem. With Claude, it's just step one of the climb. As soon it's just step one of the climb. As soon it's just step one of the climb. As soon as Claude has a number that it can beat, as Claude has a number that it can beat, as Claude has a number that it can beat, it can start optimizing. This meant that it can start optimizing. This meant that it can start optimizing. This meant that the highest leverage thing they could do the highest leverage thing they could do the highest leverage thing they could do is find more things to measure. This is is find more things to measure. This is is find more things to measure. This is why I was excited to read this article. why I was excited to read this article. why I was excited to read this article. I want to see the new novel ways they I want to see the new novel ways they I want to see the new novel ways they found to measure things. The loop thread found to measure things. The loop thread found to measure things. The loop thread by thread. This is a little cringe. Let by thread. This is a little cringe. Let by thread. This is a little cringe. Let me All of this ran in the same Slack me All of this ran in the same Slack me All of this ran in the same Slack channel with multiple engineers and channel with multiple engineers and channel with multiple engineers and Claude jamming in every thread. From Claude jamming in every thread. From Claude jamming in every thread. From there, the sprint settled into a loop.

there, the sprint settled into a loop. there, the sprint settled into a loop. One, someone would open a thread about a One, someone would open a thread about a One, someone would open a thread about a slow stretch of the journey, often with slow stretch of the journey, often with slow stretch of the journey, often with a screenshot or recording. Two, Claude a screenshot or recording. Two, Claude a screenshot or recording. Two, Claude would trace the flow and then find or would trace the flow and then find or would trace the flow and then find or build benchmarks that demonstrate the build benchmarks that demonstrate the build benchmarks that demonstrate the problem. Three, once it had promising problem. Three, once it had promising problem. Three, once it had promising results in the lab, Claude would come results in the lab, Claude would come results in the lab, Claude would come back with a PR or sometimes several back with a PR or sometimes several back with a PR or sometimes several sized more mdash is great. Once it had a sized more mdash is great. Once it had a sized more mdash is great. Once it had a promising result in the lab, Cloud would promising result in the lab, Cloud would promising result in the lab, Cloud would come back with a PR, often several sized come back with a PR, often several sized come back with a PR, often several sized for risk and review with anything user for risk and review with anything user for risk and review with anything user visible behind a flag. After it shipped, visible behind a flag. After it shipped, visible behind a flag. After it shipped, Claude watched the deploy and read the Claude watched the deploy and read the Claude watched the deploy and read the field data. If performance improved, field data. If performance improved, field data. If performance improved, Claude locked in the win by ratcheting Claude locked in the win by ratcheting Claude locked in the win by ratcheting the benchmark down. If not, it turned the benchmark down. If not, it turned the benchmark down. If not, it turned the flag off and iterated accordingly. the flag off and iterated accordingly. the flag off and iterated accordingly. Then it went looking for the next slow Then it went looking for the next slow Then it went looking for the next slow spot in the same journey. So I I'm split spot in the same journey. So I I'm split spot in the same journey. So I I'm split on this flow. On one hand, nothing on this flow. On one hand, nothing on this flow. On one hand, nothing matters if it doesn't actually improve matters if it doesn't actually improve matters if it doesn't actually improve performance for real world users. On the performance for real world users. On the performance for real world users. On the other hand, if you're not testing these other hand, if you're not testing these other hand, if you're not testing these things yourself manually as the engineer things yourself manually as the engineer things yourself manually as the engineer prompting for the changes and merging prompting for the changes and merging prompting for the changes and merging the PRs, you're not going to notice the the PRs, you're not going to notice the the PRs, you're not going to notice the types of regressions that users won't types of regressions that users won't types of regressions that users won't report. Like, let's be real here. How report. Like, let's be real here. How report. Like, let's be real here. How many users would have actually reported many users would have actually reported many users would have actually reported the sidebar issues I just found in the sidebar issues I just found in the sidebar issues I just found in Showcase? Like, who is using Claude in Showcase? Like, who is using Claude in Showcase? Like, who is using Claude in two browsers enough to notice these two browsers enough to notice these two browsers enough to notice these regressions? And who would be annoyed regressions? And who would be annoyed regressions? And who would be annoyed enough to bother like reporting that to enough to bother like reporting that to enough to bother like reporting that to Anthropic in a way that they would even Anthropic in a way that they would even Anthropic in a way that they would even notice? If they're not testing it notice? If they're not testing it notice? If they're not testing it themselves, you're going to have these themselves, you're going to have these themselves, you're going to have these types of regressions. That's why I put types of regressions. That's why I put types of regressions. That's why I put so much work in T3 code to make it so much work in T3 code to make it so much work in T3 code to make it easier for us as the T3 code developers easier for us as the T3 code developers easier for us as the T3 code developers to spin up builds to test ourselves. For to spin up builds to test ourselves. For to spin up builds to test ourselves. For example, I added a label on GitHub that example, I added a label on GitHub that example, I added a label on GitHub that I can apply to a PR that will trigger a I can apply to a PR that will trigger a I can apply to a PR that will trigger a manual build of the desktop app for Mac

manual build of the desktop app for Mac manual build of the desktop app for Mac OS. So, I can just download the DMG and OS. So, I can just download the DMG and OS. So, I can just download the DMG and test it directly without having to like test it directly without having to like test it directly without having to like fetch the branch, build it locally, and fetch the branch, build it locally, and fetch the branch, build it locally, and spin it up. This means I can even take spin it up. This means I can even take spin it up. This means I can even take one of my other machines like my MacBook one of my other machines like my MacBook one of my other machines like my MacBook Air I use for testing and go to GitHub, Air I use for testing and go to GitHub, Air I use for testing and go to GitHub, download the DMG and test it directly download the DMG and test it directly download the DMG and test it directly there without it affecting my machine. I there without it affecting my machine. I there without it affecting my machine. I think these details are just as think these details are just as think these details are just as important if not more so than measuring important if not more so than measuring important if not more so than measuring realworld user performance. And I'm realworld user performance. And I'm realworld user performance. And I'm amazed how few people have flows for amazed how few people have flows for amazed how few people have flows for this. I've talked to everyone from like this. I've talked to everyone from like this. I've talked to everyone from like small startups to independent devs to small startups to independent devs to small startups to independent devs to big businesses that are just outright big businesses that are just outright big businesses that are just outright failing in this way. Like if they want failing in this way. Like if they want failing in this way. Like if they want to test a change in a PR, that's a 15 to to test a change in a PR, that's a 15 to to test a change in a PR, that's a 15 to 25 minute process and it's so 25 minute process and it's so 25 minute process and it's so frustrating. If you're going to put slop frustrating. If you're going to put slop frustrating. If you're going to put slop somewhere, it should be there. Slop up somewhere, it should be there. Slop up somewhere, it should be there. Slop up ways to test your changes yourself with ways to test your changes yourself with ways to test your changes yourself with as little friction as possible because as little friction as possible because as little friction as possible because that friction makes you hesitant to do that friction makes you hesitant to do that friction makes you hesitant to do all of this in the first place. I am all of this in the first place. I am all of this in the first place. I am very hyped this made it in here. Someone very hyped this made it in here. Someone very hyped this made it in here. Someone shared a screen recording that showed shared a screen recording that showed shared a screen recording that showed sidebar rows popping in after the page sidebar rows popping in after the page sidebar rows popping in after the page loaded. Chat and co-work rows resolved loaded. Chat and co-work rows resolved loaded. Chat and co-work rows resolved at different times, making the page feel at different times, making the page feel at different times, making the page feel janky. None of our existing monitors janky. None of our existing monitors janky. None of our existing monitors detected it. The closest that we had was detected it. The closest that we had was detected it. The closest that we had was a cumulative layout shift, but each a cumulative layout shift, but each a cumulative layout shift, but each shift only scored about 0.008, well shift only scored about 0.008, well shift only scored about 0.008, well within the good threshold of 0.1. Isaac within the good threshold of 0.1. Isaac within the good threshold of 0.1. Isaac had the idea to reference the underlying had the idea to reference the underlying had the idea to reference the underlying layout instability API directly. I don't layout instability API directly. I don't layout instability API directly. I don't think you need to add a bunch of think you need to add a bunch of think you need to add a bunch of measurements for this. You just need to measurements for this. You just need to measurements for this. You just need to cache that one query. Uh like cache that one query. Uh like cache that one query. Uh like why are you hitting the like deep layout why are you hitting the like deep layout why are you hitting the like deep layout API for this? You don't need that. I'm API for this? You don't need that. I'm API for this? You don't need that. I'm very thankful they recently hired Addios

very thankful they recently hired Addios very thankful they recently hired Addios Mani from the Chrome team because he is Mani from the Chrome team because he is Mani from the Chrome team because he is going to clean up a lot of the slop that going to clean up a lot of the slop that going to clean up a lot of the slop that happened in this arc. Ah, anyways, happened in this arc. Ah, anyways, happened in this arc. Ah, anyways, apparently when they were doing that, apparently when they were doing that, apparently when they were doing that, they also added integration tests that they also added integration tests that they also added integration tests that opened the page with a populated opened the page with a populated opened the page with a populated sidebar, held the sidewars data until sidebar, held the sidewars data until sidebar, held the sidewars data until after first paint, and failed on any after first paint, and failed on any after first paint, and failed on any shift in any named region. Oh, this shift in any named region. Oh, this shift in any named region. Oh, this might be the problem actually. If the might be the problem actually. If the might be the problem actually. If the data in the browser has some things in data in the browser has some things in data in the browser has some things in it and the data from the network has it and the data from the network has it and the data from the network has different things, that will be a layout different things, that will be a layout different things, that will be a layout shift. If you have four threads cached shift. If you have four threads cached shift. If you have four threads cached locally and you have a fifth thread that locally and you have a fifth thread that locally and you have a fifth thread that it loads from the network, it will shift it loads from the network, it will shift it loads from the network, it will shift the four threads down when that fifth the four threads down when that fifth the four threads down when that fifth thread comes in. Their tests here might thread comes in. Their tests here might thread comes in. Their tests here might have actually caused the regression have actually caused the regression have actually caused the regression because doing this right would fail this because doing this right would fail this because doing this right would fail this test. I I love trying to reverse test. I I love trying to reverse test. I I love trying to reverse engineer where these failures come from. engineer where these failures come from. engineer where these failures come from. After the event deployed, Claude read After the event deployed, Claude read After the event deployed, Claude read the field data and found that 31% of web the field data and found that 31% of web the field data and found that 31% of web page loads moved something after the page loads moved something after the page loads moved something after the page was usable without any user page was usable without any user page was usable without any user interaction. From there, Claude worked interaction. From there, Claude worked interaction. From there, Claude worked through the causes by name. A header row through the causes by name. A header row through the causes by name. A header row that arrived late, a carrot that slid that arrived late, a carrot that slid that arrived late, a carrot that slid sideways once the user's name loaded, a sideways once the user's name loaded, a sideways once the user's name loaded, a list that moved when the scroll bar list that moved when the scroll bar list that moved when the scroll bar popped in. Claude fixed the top popped in. Claude fixed the top popped in. Claude fixed the top offenders as a batch. And when they were offenders as a batch. And when they were offenders as a batch. And when they were gone, it found the next batch. I do love gone, it found the next batch. I do love gone, it found the next batch. I do love these types of comparisons.

Do you see how much stuff is moving Do you see how much stuff is moving around on this left example here? Yeah, that's rough. This is one of the Yeah, that's rough. This is one of the big benefits of using cached data on big benefits of using cached data on big benefits of using cached data on page load. The catch is if the cache page load. The catch is if the cache page load. The catch is if the cache data is not accurate against the real data is not accurate against the real data is not accurate against the real world data, you're going to have some world data, you're going to have some world data, you're going to have some shift. But I think that like the chats shift. But I think that like the chats shift. But I think that like the chats and tasks shifting down one or two items and tasks shifting down one or two items and tasks shifting down one or two items because the real data loaded is not that because the real data loaded is not that because the real data loaded is not that big a deal, especially if you have a big a deal, especially if you have a big a deal, especially if you have a fade treatment on that for when that fade treatment on that for when that fade treatment on that for when that change happens. Someone in chat just change happens. Someone in chat just change happens. Someone in chat just asked a very good question. How do you asked a very good question. How do you asked a very good question. How do you avoid this type of issue without looking avoid this type of issue without looking avoid this type of issue without looking at the code for testing? I'm gonna at the code for testing? I'm gonna at the code for testing? I'm gonna phrase it back in a pretty silly way. phrase it back in a pretty silly way. phrase it back in a pretty silly way. How was I able to identify this issue How was I able to identify this issue How was I able to identify this issue with testing without ever seeing any of with testing without ever seeing any of with testing without ever seeing any of the code at all? Because I don't even the code at all? Because I don't even the code at all? Because I don't even have access to the repo where they wrote have access to the repo where they wrote have access to the repo where they wrote this. The blog post that the anthropic this. The blog post that the anthropic this. The blog post that the anthropic team wrote here is probably, I would team wrote here is probably, I would team wrote here is probably, I would guess, slightly less detailed than the guess, slightly less detailed than the guess, slightly less detailed than the responses Claude gave them in their responses Claude gave them in their responses Claude gave them in their threads in Slack. I don't know how you threads in Slack. I don't know how you threads in Slack. I don't know how you build these skills now because the way I build these skills now because the way I build these skills now because the way I built them was going deep in the details built them was going deep in the details built them was going deep in the details to fix all of these things myself by to fix all of these things myself by to fix all of these things myself by hand for almost a decade way before AI hand for almost a decade way before AI hand for almost a decade way before AI was even a thing you could use for dev.

was even a thing you could use for dev. was even a thing you could use for dev. And I am noticing these issues because And I am noticing these issues because And I am noticing these issues because of my history doing these things by hand of my history doing these things by hand of my history doing these things by hand before. I don't need to see the code or before. I don't need to see the code or before. I don't need to see the code or read the tests to know that. I can tell read the tests to know that. I can tell read the tests to know that. I can tell from the PR description or from the from the PR description or from the from the PR description or from the response Claude gives me in my Claude response Claude gives me in my Claude response Claude gives me in my Claude code or from Astra in codeex. A lot of code or from Astra in codeex. A lot of code or from Astra in codeex. A lot of this just comes down to yeah this you this just comes down to yeah this you this just comes down to yeah this you said it branch intuition and you have to said it branch intuition and you have to said it branch intuition and you have to build that over time. I didn't have to build that over time. I didn't have to build that over time. I didn't have to read the code to have these potential read the code to have these potential read the code to have these potential realizations and I still could be wrong. realizations and I still could be wrong. realizations and I still could be wrong. Of course I might not be correct in that Of course I might not be correct in that Of course I might not be correct in that these measurements are enabling or these measurements are enabling or these measurements are enabling or encouraging these specific types of encouraging these specific types of encouraging these specific types of regressions. But it's a theory that if I regressions. But it's a theory that if I regressions. But it's a theory that if I had codebase access I could ask Claude had codebase access I could ask Claude had codebase access I could ask Claude to go prove out for me and then use that to go prove out for me and then use that to go prove out for me and then use that to tell the team like hey I like where to tell the team like hey I like where to tell the team like hey I like where you're going here but these tests kind you're going here but these tests kind you're going here but these tests kind of suck. here are the regressions that of suck. here are the regressions that of suck. here are the regressions that we're going to cause if we don't shift we're going to cause if we don't shift we're going to cause if we don't shift how we think about this. So yeah, how we think about this. So yeah, how we think about this. So yeah, reading the code does not solve this reading the code does not solve this reading the code does not solve this problem at all. You got to read the problem at all. You got to read the problem at all. You got to read the responses from claude and understand responses from claude and understand responses from claude and understand what it is doing when it does it. If what it is doing when it does it. If what it is doing when it does it. If you're actually reading all the code you're actually reading all the code you're actually reading all the code when you do these types of tests, you're when you do these types of tests, you're when you do these types of tests, you're not going to get very far. Because I'll not going to get very far. Because I'll not going to get very far. Because I'll be frank, when I was doing all the data be frank, when I was doing all the data be frank, when I was doing all the data loading stuff in the T3 code like loading stuff in the T3 code like loading stuff in the T3 code like overhaul, I probably wrote a thousand overhaul, I probably wrote a thousand overhaul, I probably wrote a thousand lines of tests in slop for lines of tests in slop for lines of tests in slop for validating my findings for every 10 validating my findings for every 10 validating my findings for every 10 lines of actual code I wrote. It's lines of actual code I wrote. It's lines of actual code I wrote. It's probably a 100 to1 ratio. And the code I probably a 100 to1 ratio. And the code I probably a 100 to1 ratio. And the code I wrote that actually shipped would have wrote that actually shipped would have wrote that actually shipped would have been worse if I hadn't taken the been worse if I hadn't taken the been worse if I hadn't taken the opportunity to slop a shitload of tests opportunity to slop a shitload of tests opportunity to slop a shitload of tests to verify my changes. And as far as I to verify my changes. And as far as I to verify my changes. And as far as I know, there were no regressions from know, there were no regressions from know, there were no regressions from those overhauls, which is actually those overhauls, which is actually those overhauls, which is actually surprising to me. I expected it to be surprising to me. I expected it to be surprising to me. I expected it to be pretty brutal and wasn't at all. So pretty brutal and wasn't at all. So pretty brutal and wasn't at all. So yeah, take it as you will. I don't know

yeah, take it as you will. I don't know yeah, take it as you will. I don't know how you build this intuition now. Maybe how you build this intuition now. Maybe how you build this intuition now. Maybe you just ask cloud questions. I don't you just ask cloud questions. I don't you just ask cloud questions. I don't know. It's worth hiring somebody who has know. It's worth hiring somebody who has know. It's worth hiring somebody who has it if you don't have someone on your it if you don't have someone on your it if you don't have someone on your team with it and you care about these team with it and you care about these team with it and you care about these things and it's probably learnable things and it's probably learnable things and it's probably learnable still. I just don't know how I'd still. I just don't know how I'd still. I just don't know how I'd recommend to do it. The next chunk of recommend to do it. The next chunk of recommend to do it. The next chunk of this blog was actually one of the more this blog was actually one of the more this blog was actually one of the more fun parts. Scaling horizontally. Once fun parts. Scaling horizontally. Once fun parts. Scaling horizontally. Once the loop worked on one thread, running the loop worked on one thread, running the loop worked on one thread, running it on more was just a matter of opening it on more was just a matter of opening it on more was just a matter of opening them. It's one of the really fun things them. It's one of the really fun things them. It's one of the really fun things about setting up a cloud environment or, about setting up a cloud environment or, about setting up a cloud environment or, I don't know, using something like T3 I don't know, using something like T3 I don't know, using something like T3 Code and the remote machine management Code and the remote machine management Code and the remote machine management stuff. Once you have an end-to-end flow stuff. Once you have an end-to-end flow stuff. Once you have an end-to-end flow that is working for figuring out a class that is working for figuring out a class that is working for figuring out a class of issue or bug, you can spin up a bunch of issue or bug, you can spin up a bunch of issue or bug, you can spin up a bunch of threads on a bunch of instances on a of threads on a bunch of instances on a of threads on a bunch of instances on a bunch of machines and immediately bunch of machines and immediately bunch of machines and immediately parallelize it. We're all hopefully parallelize it. We're all hopefully parallelize it. We're all hopefully engineers and we've experienced this engineers and we've experienced this engineers and we've experienced this before. Writing code to do a task once before. Writing code to do a task once before. Writing code to do a task once is rarely worth it. But once you wrote is rarely worth it. But once you wrote is rarely worth it. But once you wrote the code, which might take hours, even the code, which might take hours, even the code, which might take hours, even if the task took five minutes, once if the task took five minutes, once if the task took five minutes, once you've written all that code, you can you've written all that code, you can you've written all that code, you can now run it again and again and again. now run it again and again and again. now run it again and again and again. You can parallelize it. It can do a You can parallelize it. It can do a You can parallelize it. It can do a bunch of and if it turns that bunch of and if it turns that bunch of and if it turns that 5minute thing into 3 seconds and you can 5minute thing into 3 seconds and you can 5minute thing into 3 seconds and you can run hundreds of them at once, it run hundreds of them at once, it run hundreds of them at once, it fundamentally changes the way you fundamentally changes the way you fundamentally changes the way you operate with that thing. So yeah, sure I operate with that thing. So yeah, sure I operate with that thing. So yeah, sure I put a lot of effort into the testing I put a lot of effort into the testing I put a lot of effort into the testing I built for my performance improvement built for my performance improvement built for my performance improvement changes, but once I had built all of changes, but once I had built all of changes, but once I had built all of that testing, it was trivial to fan out that testing, it was trivial to fan out that testing, it was trivial to fan out lots of agents with lots of theories to lots of agents with lots of theories to lots of agents with lots of theories to try and solve lots of things. And the try and solve lots of things. And the try and solve lots of things. And the result is that my agent automerged like result is that my agent automerged like result is that my agent automerged like 40 PRs improving performance in T3 code.

40 PRs improving performance in T3 code. 40 PRs improving performance in T3 code. And the only real regressions were one And the only real regressions were one And the only real regressions were one change to animations on the actual change to animations on the actual change to animations on the actual desktop that people didn't like, which I desktop that people didn't like, which I desktop that people didn't like, which I understand. I didn't like the change understand. I didn't like the change understand. I didn't like the change either. And it broke the sliding ticker either. And it broke the sliding ticker either. And it broke the sliding ticker for tweets on the homepage of the for tweets on the homepage of the for tweets on the homepage of the marketing site. But other than that, marketing site. But other than that, marketing site. But other than that, even Astra and its weird spikiness. 38 even Astra and its weird spikiness. 38 even Astra and its weird spikiness. 38 out of the 40 PRs, it merged itself. out of the 40 PRs, it merged itself. out of the 40 PRs, it merged itself. Real performance wins. Pretty cool. Once Real performance wins. Pretty cool. Once Real performance wins. Pretty cool. Once you have the testing all set up, this is you have the testing all set up, this is you have the testing all set up, this is an interesting detail, though. Instead an interesting detail, though. Instead an interesting detail, though. Instead of closing threads once the original of closing threads once the original of closing threads once the original request had been fulfilled, Claude would request had been fulfilled, Claude would request had been fulfilled, Claude would keep going. An individual thread would keep going. An individual thread would keep going. An individual thread would put up 50, sometimes a hundred put up 50, sometimes a hundred put up 50, sometimes a hundred optimization PRs. Interestingly, it was optimization PRs. Interestingly, it was optimization PRs. Interestingly, it was Claude, not one of us, opening new Claude, not one of us, opening new Claude, not one of us, opening new threads to chase opportunities it had threads to chase opportunities it had threads to chase opportunities it had found on its own as parts of separate found on its own as parts of separate found on its own as parts of separate investigations or nightly jobs. Shelley, investigations or nightly jobs. Shelley, investigations or nightly jobs. Shelley, one of the engineers in the channel, one of the engineers in the channel, one of the engineers in the channel, observed, quote, "The model is a numbers observed, quote, "The model is a numbers observed, quote, "The model is a numbers demon." Yeah, it's surprisingly easy to demon." Yeah, it's surprisingly easy to demon." Yeah, it's surprisingly easy to set something like this up. Every set something like this up. Every set something like this up. Every measurement found something to improve. measurement found something to improve. measurement found something to improve. Claude ran a React hook census and found Claude ran a React hook census and found Claude ran a React hook census and found 6,900 hooks in 900 store subscriptions 6,900 hooks in 900 store subscriptions 6,900 hooks in 900 store subscriptions in the composer's typing path, in the composer's typing path, in the composer's typing path, rerendering on every keystroke. rerendering on every keystroke. rerendering on every keystroke. [groaning] [groaning] [groaning] Nearly 7,000 hooks for the composer.

Nearly 7,000 hooks for the composer. Nearly 7,000 hooks for the composer. This is what h maybe we should read the This is what h maybe we should read the This is what h maybe we should read the code, guys. I I take it all back. code, guys. I I take it all back. code, guys. I I take it all back. Anyways, Claude counted style Anyways, Claude counted style Anyways, Claude counted style recalculations and found a single root recalculations and found a single root recalculations and found a single root has selector that added 24 milliseconds has selector that added 24 milliseconds has selector that added 24 milliseconds to every single DOM change. You know how to every single DOM change. You know how to every single DOM change. You know how many of these types of things there are? many of these types of things there are? many of these types of things there are? Like one simple CSS thing causes every Like one simple CSS thing causes every Like one simple CSS thing causes every change in the browser, even in Electron, change in the browser, even in Electron, change in the browser, even in Electron, to be way slower. It's way too easy to to be way slower. It's way too easy to to be way slower. It's way too easy to cause these things. And Chrome's tools cause these things. And Chrome's tools cause these things. And Chrome's tools do not make it easy enough to find these do not make it easy enough to find these do not make it easy enough to find these things, sadly, especially GPU stuff. things, sadly, especially GPU stuff. things, sadly, especially GPU stuff. Chrome's okay at helping you with CPU Chrome's okay at helping you with CPU Chrome's okay at helping you with CPU stuff, at event loop stuff, anything stuff, at event loop stuff, anything stuff, at event loop stuff, anything JavaScript side. Once you're in the GPU JavaScript side. Once you're in the GPU JavaScript side. Once you're in the GPU accelerated CSS layer, good luck. Have accelerated CSS layer, good luck. Have accelerated CSS layer, good luck. Have fun. I'm going to choose a different fun. I'm going to choose a different fun. I'm going to choose a different mindset for how I'm going to approach mindset for how I'm going to approach mindset for how I'm going to approach anthropic engineering going forward. anthropic engineering going forward. anthropic engineering going forward. Now, this might get me in trouble. I Now, this might get me in trouble. I Now, this might get me in trouble. I don't care. Anthropic has chosen to be don't care. Anthropic has chosen to be don't care. Anthropic has chosen to be the first company to really really the first company to really really the first company to really really experiment on what happens if you let AI experiment on what happens if you let AI experiment on what happens if you let AI write too much of the code. Since they write too much of the code. Since they write too much of the code. Since they did this in the sonnet 4 days, they have did this in the sonnet 4 days, they have did this in the sonnet 4 days, they have a code base that is full of slop and a code base that is full of slop and a code base that is full of slop and garbage. They learned a lot of lessons.

garbage. They learned a lot of lessons. garbage. They learned a lot of lessons. They made the model much better. And now They made the model much better. And now They made the model much better. And now for the betterment of everyone else, for the betterment of everyone else, for the betterment of everyone else, they are using the models to clean up they are using the models to clean up they are using the models to clean up the slop that they made themselves. The the slop that they made themselves. The the slop that they made themselves. The cla site and the cloud desktop app and cla site and the cloud desktop app and cla site and the cloud desktop app and I'd argue even cla code itself are them I'd argue even cla code itself are them I'd argue even cla code itself are them dog fooding both how much slop can dog fooding both how much slop can dog fooding both how much slop can something take and also when are the something take and also when are the something take and also when are the models good enough to clean up the mess models good enough to clean up the mess models good enough to clean up the mess that we created. They also found a code that we created. They also found a code that we created. They also found a code path that had a leftover location. Which path that had a leftover location. Which path that had a leftover location. Which is like a browser refresh that would is like a browser refresh that would is like a browser refresh that would cause half a million hidden reloads a cause half a million hidden reloads a cause half a million hidden reloads a day that none of their load metrics day that none of their load metrics day that none of their load metrics could see. Claude read profiler samples could see. Claude read profiler samples could see. Claude read profiler samples from idle tabs and found identical cache from idle tabs and found identical cache from idle tabs and found identical cache snapshots being cloned into index DB snapshots being cloned into index DB snapshots being cloned into index DB twice every minute all on the main twice every minute all on the main twice every minute all on the main thread. Oof. This actually might also be thread. Oof. This actually might also be thread. Oof. This actually might also be the change I was talking about with the the change I was talking about with the the change I was talking about with the sidebar issues. If they made it so that sidebar issues. If they made it so that sidebar issues. If they made it so that loading data from the network doesn't loading data from the network doesn't loading data from the network doesn't write to the index DB storage as much, write to the index DB storage as much, write to the index DB storage as much, that would prevent that storage from that would prevent that storage from that would prevent that storage from being updated. Great point from Jamie being updated. Great point from Jamie being updated. Great point from Jamie from Convex here. Companies that are from Convex here. Companies that are from Convex here. Companies that are good at making models are not good at making models are not good at making models are not necessarily good at software necessarily good at software necessarily good at software engineering. This is a lesson playing engineering. This is a lesson playing engineering. This is a lesson playing out slow as I'll go even further out slow as I'll go even further out slow as I'll go even further than this, Jamie. There was a deep, as than this, Jamie. There was a deep, as than this, Jamie. There was a deep, as I'm sure a lot of y'all know, a deep I'm sure a lot of y'all know, a deep I'm sure a lot of y'all know, a deep rivalry between OpenAI and Anthropic rivalry between OpenAI and Anthropic rivalry between OpenAI and Anthropic because Anthropic was largely started because Anthropic was largely started because Anthropic was largely started from the original pre-training team at from the original pre-training team at from the original pre-training team at OpenAI. Daario and friends were all OpenAI. Daario and friends were all OpenAI. Daario and friends were all helping build GPT3 and they left because helping build GPT3 and they left because helping build GPT3 and they left because of a lack of alignment between them and of a lack of alignment between them and of a lack of alignment between them and the leaders at OpenAI. They still say a the leaders at OpenAI. They still say a the leaders at OpenAI. They still say a lot of this is around safety concerns. I lot of this is around safety concerns. I lot of this is around safety concerns. I don't actually think they were that don't actually think they were that don't actually think they were that concerned about safety in the GBT3 days.

concerned about safety in the GBT3 days. concerned about safety in the GBT3 days. I and many others have started believing I and many others have started believing I and many others have started believing the real friction here was the the real friction here was the the real friction here was the difference in engineering versus difference in engineering versus difference in engineering versus research where OpenAI despite being a research where OpenAI despite being a research where OpenAI despite being a research company was led by two research company was led by two research company was led by two engineers Sam Alman and GDB which meant engineers Sam Alman and GDB which meant engineers Sam Alman and GDB which meant that engineers were the ones commanding that engineers were the ones commanding that engineers were the ones commanding researchers to go do research stuff and researchers to go do research stuff and researchers to go do research stuff and that meant they were holding them to that meant they were holding them to that meant they were holding them to engineering standards. because they were engineering standards. because they were engineering standards. because they were expecting engineering deadlines and expecting engineering deadlines and expecting engineering deadlines and timelines and all of that type of stuff. timelines and all of that type of stuff. timelines and all of that type of stuff. And that caused disdain between the two And that caused disdain between the two And that caused disdain between the two groups. Since those guys left, it groups. Since those guys left, it groups. Since those guys left, it definitely caused some frustration and definitely caused some frustration and definitely caused some frustration and open AAI towards researchers and they open AAI towards researchers and they open AAI towards researchers and they started looking for ways to engineer started looking for ways to engineer started looking for ways to engineer solutions to research problems. solutions to research problems. solutions to research problems. Meanwhile, at Enthropic, it meant that Meanwhile, at Enthropic, it meant that Meanwhile, at Enthropic, it meant that they kind of didn't like software they kind of didn't like software they kind of didn't like software engineers that much. And you can still engineers that much. And you can still engineers that much. And you can still see that in their coms even now. I think see that in their coms even now. I think see that in their coms even now. I think a lot of why Anthropic pushed so hard a lot of why Anthropic pushed so hard a lot of why Anthropic pushed so hard towards this idea of what if we replace towards this idea of what if we replace towards this idea of what if we replace all the engineers with AI is because all the engineers with AI is because all the engineers with AI is because they didn't like engineers, which means they didn't like engineers, which means they didn't like engineers, which means they probably weren't hiring the best they probably weren't hiring the best they probably weren't hiring the best ones. And I'm going to be very real ones. And I'm going to be very real ones. And I'm going to be very real here. It's some of the advice I give the here. It's some of the advice I give the here. It's some of the advice I give the most. Hire for what you love, not what most. Hire for what you love, not what most. Hire for what you love, not what you hate, because you'll make much you hate, because you'll make much you hate, because you'll make much better hiring decisions. Since they were better hiring decisions. Since they were better hiring decisions. Since they were hiring engineers and they don't like hiring engineers and they don't like hiring engineers and they don't like engineers, they weren't making great engineers, they weren't making great engineers, they weren't making great engineering hiring decisions. It's engineering hiring decisions. It's engineering hiring decisions. It's difficult to automate things you don't difficult to automate things you don't difficult to automate things you don't understand, and it's difficult to understand, and it's difficult to understand, and it's difficult to understand something that you don't understand something that you don't understand something that you don't respect. Absolutely agree, Jamie. Back respect. Absolutely agree, Jamie. Back respect. Absolutely agree, Jamie. Back to the performance improvements because to the performance improvements because to the performance improvements because I I love this stuff. We rarely knew I I love this stuff. We rarely knew I I love this stuff. We rarely knew where a thread would lead. In a sweep where a thread would lead. In a sweep where a thread would lead. In a sweep for CPU hitches, Claude noticed that for CPU hitches, Claude noticed that for CPU hitches, Claude noticed that highlighting a finished code block could highlighting a finished code block could highlighting a finished code block could freeze the page for about a second. Oof.

freeze the page for about a second. Oof. freeze the page for about a second. Oof. Syntax highlighting in the browser. Syntax highlighting in the browser. Syntax highlighting in the browser. Synthe general is one of those like Synthe general is one of those like Synthe general is one of those like really annoying problems to get right. really annoying problems to get right. really annoying problems to get right. It dug in the lab and found the culprit. It dug in the lab and found the culprit. It dug in the lab and found the culprit. M dashes. Oh man. Oh man. I I saw M dashes. Oh man. Oh man. I I saw M dashes. Oh man. Oh man. I I saw someone say in chat that there was an M someone say in chat that there was an M someone say in chat that there was an M dashes section. I I was not prepared for dashes section. I I was not prepared for dashes section. I I was not prepared for replies markdown contained any non-Latin replies markdown contained any non-Latin replies markdown contained any non-Latin one character like an M dash or a curly one character like an M dash or a curly one character like an M dash or a curly quote. V8 would store the entire string quote. V8 would store the entire string quote. V8 would store the entire string as UTF-16 which put every syntax as UTF-16 which put every syntax as UTF-16 which put every syntax highlighting reax on a slower two byte highlighting reax on a slower two byte highlighting reax on a slower two byte path with a 20line change to copy each path with a 20line change to copy each path with a 20line change to copy each code block into a one byte string before code block into a one byte string before code block into a one byte string before highlighting it. Blocking main thread highlighting it. Blocking main thread highlighting it. Blocking main thread for 0.35 seconds is in my opinion always for 0.35 seconds is in my opinion always for 0.35 seconds is in my opinion always unacceptable. We moved our syntax unacceptable. We moved our syntax unacceptable. We moved our syntax highlighting off to WASOM because since highlighting off to WASOM because since highlighting off to WASOM because since it now runs in a worker, it won't block it now runs in a worker, it won't block it now runs in a worker, it won't block main thread at all, which is quite nice. main thread at all, which is quite nice. main thread at all, which is quite nice. Write me five code blocks in various Write me five code blocks in various Write me five code blocks in various languages doing some simple demo that's languages doing some simple demo that's languages doing some simple demo that's at least 10 lines of code and I'll put at least 10 lines of code and I'll put at least 10 lines of code and I'll put them in the chat here so I can see them.

You might notice the fade. I added the You might notice the fade. I added the fade to hide the fact that I was doing fade to hide the fact that I was doing fade to hide the fact that I was doing the actual highlighting in a different the actual highlighting in a different the actual highlighting in a different thread to not block the main thread. So thread to not block the main thread. So thread to not block the main thread. So you can still navigate around T3 code you can still navigate around T3 code you can still navigate around T3 code and it still flies even when those and it still flies even when those and it still flies even when those things are happening in the background. things are happening in the background. things are happening in the background. It's able to syntax highlight various It's able to syntax highlight various It's able to syntax highlight various different languages all in a row while different languages all in a row while different languages all in a row while you have a bunch of other stuff going you have a bunch of other stuff going you have a bunch of other stuff going without it affecting the main thread's without it affecting the main thread's without it affecting the main thread's performance at all. So yeah, they should performance at all. So yeah, they should performance at all. So yeah, they should probably be doing that too. The catch probably be doing that too. The catch probably be doing that too. The catch here is that you have to load a bigger here is that you have to load a bigger here is that you have to load a bigger bundle for that syntax highlighting, but bundle for that syntax highlighting, but bundle for that syntax highlighting, but if you lazy load it anyways, it comes in if you lazy load it anyways, it comes in if you lazy load it anyways, it comes in later once and you're fine. So yeah, I'm later once and you're fine. So yeah, I'm later once and you're fine. So yeah, I'm surprised that they didn't get Claude to surprised that they didn't get Claude to surprised that they didn't get Claude to propose that to them because it would propose that to them because it would propose that to them because it would have helped with a lot of these have helped with a lot of these have helped with a lot of these problems. I would argue that since we're problems. I would argue that since we're problems. I would argue that since we're not reading the code anyways, main not reading the code anyways, main not reading the code anyways, main thread being blocked at all for syntax thread being blocked at all for syntax thread being blocked at all for syntax highlighting is bad. Apparently, by the highlighting is bad. Apparently, by the highlighting is bad. Apparently, by the second week, they could barely even second week, they could barely even second week, they could barely even summarize their output into daily summarize their output into daily summarize their output into daily updates. On the busiest days, more than updates. On the busiest days, more than updates. On the busiest days, more than 200 changes would land. Claude kept 200 changes would land. Claude kept 200 changes would land. Claude kept proposing new benchmarks. About a third proposing new benchmarks. About a third proposing new benchmarks. About a third of PRs included additional telemetry or of PRs included additional telemetry or of PRs included additional telemetry or guardrails, and each new instrument guardrails, and each new instrument guardrails, and each new instrument generated more threads with more generated more threads with more generated more threads with more opportunities. Working in one channel opportunities. Working in one channel opportunities. Working in one channel meant everything happened in the open.

meant everything happened in the open. meant everything happened in the open. We jumped in and out of each other's We jumped in and out of each other's We jumped in and out of each other's threads to debate decisions and threads to debate decisions and threads to debate decisions and celebrate wins. Word spread. Other teams celebrate wins. Word spread. Other teams celebrate wins. Word spread. Other teams started to bring their changes into the started to bring their changes into the started to bring their changes into the channel to have them reviewed for channel to have them reviewed for channel to have them reviewed for performance. New projects were written performance. New projects were written performance. New projects were written in subtly more performant ways because in subtly more performant ways because in subtly more performant ways because of the guard rails that had been added of the guard rails that had been added of the guard rails that had been added and the new cloud skills that had been and the new cloud skills that had been and the new cloud skills that had been introduced. I do really like this idea introduced. I do really like this idea introduced. I do really like this idea of gang prompting at companies, like of gang prompting at companies, like of gang prompting at companies, like having a channel that can be used by having a channel that can be used by having a channel that can be used by various different people across various various different people across various various different people across various different teams with shared context and different teams with shared context and different teams with shared context and capability. This isn't reading like an capability. This isn't reading like an capability. This isn't reading like an ad for claude tag in Claude and Slack, ad for claude tag in Claude and Slack, ad for claude tag in Claude and Slack, but it is a very compelling one. It but it is a very compelling one. It but it is a very compelling one. It makes me want to go set up something makes me want to go set up something makes me want to go set up something similar for us with all the T3 codeb similar for us with all the T3 codeb similar for us with all the T3 codeb stuff that Maria's been doing. And now stuff that Maria's been doing. And now stuff that Maria's been doing. And now we are in the guard rails. We' prepared we are in the guard rails. We' prepared we are in the guard rails. We' prepared for the pace because nearly everything for the pace because nearly everything for the pace because nearly everything we touched was a hot path. the first we touched was a hot path. the first we touched was a hot path. the first paint, composer, transcript, etc. We'd paint, composer, transcript, etc. We'd paint, composer, transcript, etc. We'd established our safety mechanisms up established our safety mechanisms up established our safety mechanisms up front. Every PR went through automated front. Every PR went through automated front. Every PR went through automated reviews with at least one human reviews with at least one human reviews with at least one human approval. Unit tests always came before approval. Unit tests always came before approval. Unit tests always came before optimizations, and anything that could optimizations, and anything that could optimizations, and anything that could cause a user visible problem shipped cause a user visible problem shipped cause a user visible problem shipped behind a short-lived feature flag. I behind a short-lived feature flag. I behind a short-lived feature flag. I don't believe you there. Regardless, don't believe you there. Regardless, don't believe you there. Regardless, once the flag started to pile up, they once the flag started to pile up, they once the flag started to pile up, they opened a thread to coordinate their opened a thread to coordinate their opened a thread to coordinate their rollout and cleanup. Claude classified rollout and cleanup. Claude classified rollout and cleanup. Claude classified every flag as a kill switch or ramp and every flag as a kill switch or ramp and every flag as a kill switch or ramp and retired each one as soon as it was safe.

retired each one as soon as it was safe. retired each one as soon as it was safe. Across the two weeks, they introduced Across the two weeks, they introduced Across the two weeks, they introduced nearly 200 flags, more than half of nearly 200 flags, more than half of nearly 200 flags, more than half of which were already cleaned up by the which were already cleaned up by the which were already cleaned up by the end. That is nice. Cleaning up the slop end. That is nice. Cleaning up the slop end. That is nice. Cleaning up the slop as you add it is an underrated thing. as you add it is an underrated thing. as you add it is an underrated thing. New performance wins decay in a New performance wins decay in a New performance wins decay in a fast-moving codebase and code chips fast fast-moving codebase and code chips fast fast-moving codebase and code chips fast and anthropic. Once a project proved a and anthropic. Once a project proved a and anthropic. Once a project proved a win, they invested in ways to protect win, they invested in ways to protect win, they invested in ways to protect it. Static Composer, for example, is it. Static Composer, for example, is it. Static Composer, for example, is brittle by design. We show users an HML brittle by design. We show users an HML brittle by design. We show users an HML copy of the page almost immediately, and copy of the page almost immediately, and copy of the page almost immediately, and we let React paint directly on top we let React paint directly on top we let React paint directly on top after. If the React render is off by after. If the React render is off by after. If the React render is off by even a pixel, all of the magic is lost. even a pixel, all of the magic is lost. even a pixel, all of the magic is lost. So Claude built dozens of guardrails. So Claude built dozens of guardrails. So Claude built dozens of guardrails. Static markup is generated by rendering Static markup is generated by rendering Static markup is generated by rendering the real React component in JSDOM and the real React component in JSDOM and the real React component in JSDOM and tests guarantee that they never drift. tests guarantee that they never drift. tests guarantee that they never drift. Integration test suite compares the Integration test suite compares the Integration test suite compares the static page against the React render static page against the React render static page against the React render across 14 different viewport sizes and across 14 different viewport sizes and across 14 different viewport sizes and then asserts alignment within one pixel. then asserts alignment within one pixel. then asserts alignment within one pixel. A keystroke test types straight through A keystroke test types straight through A keystroke test types straight through the handoff and fails on any lost the handoff and fails on any lost the handoff and fails on any lost reordered keys. And in the field, every reordered keys. And in the field, every reordered keys. And in the field, every handoff report shifts to a tenth of a handoff report shifts to a tenth of a handoff report shifts to a tenth of a pixel. Cloud opens a thread for any pixel. Cloud opens a thread for any pixel. Cloud opens a thread for any event with nonzero movement. interesting event with nonzero movement. interesting event with nonzero movement. interesting that they have like threads auto open that they have like threads auto open that they have like threads auto open when users experience things by the when users experience things by the when users experience things by the sound of that. Not everything could be sound of that. Not everything could be sound of that. Not everything could be caught in the lab though, so they had to caught in the lab though, so they had to caught in the lab though, so they had to make use of the oldest guardrail in the make use of the oldest guardrail in the make use of the oldest guardrail in the book, incremental rollouts. High-risk book, incremental rollouts. High-risk book, incremental rollouts. High-risk changes were rolled out to employees changes were rolled out to employees changes were rolled out to employees first, then 1% of users, then everyone.

first, then 1% of users, then everyone. first, then 1% of users, then everyone. For hours after we released the static For hours after we released the static For hours after we released the static composer internally, a teammate shared a composer internally, a teammate shared a composer internally, a teammate shared a screen recording of a layout shift that screen recording of a layout shift that screen recording of a layout shift that none of our metrics could see. When he none of our metrics could see. When he none of our metrics could see. When he opened cloudi in a new tab, the composer opened cloudi in a new tab, the composer opened cloudi in a new tab, the composer would drop mdash, but it wasn't our would drop mdash, but it wasn't our would drop mdash, but it wasn't our code. This is a Chrome extension I'm code. This is a Chrome extension I'm code. This is a Chrome extension I'm calling it now. Chrome resizing the calling it now. Chrome resizing the calling it now. Chrome resizing the page, not the handoff from the static page, not the handoff from the static page, not the handoff from the static composer. On the new tab page, Chrome composer. On the new tab page, Chrome composer. On the new tab page, Chrome draws its own 56 pixel footer below the draws its own 56 pixel footer below the draws its own 56 pixel footer below the page managed by enthropic.com customized page managed by enthropic.com customized page managed by enthropic.com customized Chrome. When the tab navigates to Claude Chrome. When the tab navigates to Claude Chrome. When the tab navigates to Claude AI, the footer goes away and the page AI, the footer goes away and the page AI, the footer goes away and the page gets 56 pixels taller, but only 100 gets 56 pixels taller, but only 100 gets 56 pixels taller, but only 100 milliseconds after the first paint. milliseconds after the first paint. milliseconds after the first paint. Okay, so it wasn't a Chrome extension, Okay, so it wasn't a Chrome extension, Okay, so it wasn't a Chrome extension, which is the cause of most of my weird which is the cause of most of my weird which is the cause of most of my weird hellish issues other than Brave. Brave hellish issues other than Brave. Brave hellish issues other than Brave. Brave causes so many problems for us. That all causes so many problems for us. That all causes so many problems for us. That all aside, this is weird managed profile aside, this is weird managed profile aside, this is weird managed profile stuff that causes niche bugs within stuff that causes niche bugs within stuff that causes niche bugs within Chrome's rendering itself. They are Chrome's rendering itself. They are Chrome's rendering itself. They are putting the Oh, this is actually putting the Oh, this is actually putting the Oh, this is actually probably the bigger problem. They are probably the bigger problem. They are probably the bigger problem. They are rendering the composer based on a rendering the composer based on a rendering the composer based on a percentage of page height and a distance percentage of page height and a distance percentage of page height and a distance from the top rather than a fixed from the top rather than a fixed from the top rather than a fixed location on the bottom. By doing that, location on the bottom. By doing that, location on the bottom. By doing that, they need to fire a JS calculation, they need to fire a JS calculation, they need to fire a JS calculation, which means they are now reliant on the which means they are now reliant on the which means they are now reliant on the page resize behaviors. And it seems like page resize behaviors. And it seems like page resize behaviors. And it seems like that didn't or get triggered on time or that didn't or get triggered on time or that didn't or get triggered on time or worse it triggered after the 100 worse it triggered after the 100 worse it triggered after the 100 millisecond like repaint occurred or the millisecond like repaint occurred or the millisecond like repaint occurred or the 100 millcond shift and then it repainted 100 millcond shift and then it repainted 100 millcond shift and then it repainted and moved down possibly even before the and moved down possibly even before the and moved down possibly even before the JS loaded in. Fascinating. It was JS loaded in. Fascinating. It was JS loaded in. Fascinating. It was because of Chrome speculative loading.

because of Chrome speculative loading. because of Chrome speculative loading. While a user was typing the URL in the While a user was typing the URL in the While a user was typing the URL in the address bar, Chrome would pre-render the address bar, Chrome would pre-render the address bar, Chrome would pre-render the page in the background and the height of page in the background and the height of page in the background and the height of the current tab. Huh. Okay, that the current tab. Huh. Okay, that the current tab. Huh. Okay, that actually is is weird, but it makes actually is is weird, but it makes actually is is weird, but it makes sense. Yeah, when you're on this page, sense. Yeah, when you're on this page, sense. Yeah, when you're on this page, there's a certain amount of space there's a certain amount of space there's a certain amount of space available. So, if there was something on available. So, if there was something on available. So, if there was something on the bottom here that caused this window the bottom here that caused this window the bottom here that caused this window to be smaller and you type cloud.ai/ to be smaller and you type cloud.ai/ to be smaller and you type cloud.ai/ new, this is now preloading in the new, this is now preloading in the new, this is now preloading in the background and when I hit enter on it, background and when I hit enter on it, background and when I hit enter on it, it doesn't have to load as much as a it doesn't have to load as much as a it doesn't have to load as much as a result of that. But Chrome's result of that. But Chrome's result of that. But Chrome's pre-fetching triggering a pre-fetching triggering a pre-fetching triggering a behind-the-scenes render that is at a behind-the-scenes render that is at a behind-the-scenes render that is at a different page size is fascinating. And different page size is fascinating. And different page size is fascinating. And I would guess this is a nonzero part of I would guess this is a nonzero part of I would guess this is a nonzero part of why they hired our friend Adiasmani from why they hired our friend Adiasmani from why they hired our friend Adiasmani from the Chrome team. It's actually one of the Chrome team. It's actually one of the Chrome team. It's actually one of the more fun ones that they have in the more fun ones that they have in the more fun ones that they have in here. I think that was that's a really here. I think that was that's a really here. I think that was that's a really cool finding. Yeah, it's from the Chrome cool finding. Yeah, it's from the Chrome cool finding. Yeah, it's from the Chrome pre-render. If I was using Claude in pre-render. If I was using Claude in pre-render. If I was using Claude in Slack and I had a thread like this and I Slack and I had a thread like this and I Slack and I had a thread like this and I saw Claude come in and point out a saw Claude come in and point out a saw Claude come in and point out a Chrome behavior I didn't know about, I Chrome behavior I didn't know about, I Chrome behavior I didn't know about, I would be the most insufferable human for would be the most insufferable human for would be the most insufferable human for at least a few hours. I'd be like, "What at least a few hours. I'd be like, "What at least a few hours. I'd be like, "What the actual This is AGI." They also the actual This is AGI." They also the actual This is AGI." They also have a section at the end here about have a section at the end here about have a section at the end here about steering, which I think should be quite steering, which I think should be quite steering, which I think should be quite fun. The loop was productive, but it fun. The loop was productive, but it fun. The loop was productive, but it wasn't autonomous. Keeping it fast, wasn't autonomous. Keeping it fast, wasn't autonomous. Keeping it fast, safe, and on track was our job. And it safe, and on track was our job. And it safe, and on track was our job. And it had three parts. First, ambition. By had three parts. First, ambition. By had three parts. First, ambition. By default, Claude is careful about scope.

default, Claude is careful about scope. default, Claude is careful about scope. Hey chat, do y'all feel that Claude is Hey chat, do y'all feel that Claude is Hey chat, do y'all feel that Claude is by default careful with its scope? I by default careful with its scope? I by default careful with its scope? I want a yes, no, or LOL from all of want a yes, no, or LOL from all of want a yes, no, or LOL from all of y'all. Thank you guys. I think the point's been Thank you guys. I think the point's been made. Regardless, yeah, helping it know made. Regardless, yeah, helping it know made. Regardless, yeah, helping it know it can push further helps a lot. You'd it can push further helps a lot. You'd it can push further helps a lot. You'd be amazed how much like one small change be amazed how much like one small change be amazed how much like one small change to your claude MD will affect these to your claude MD will affect these to your claude MD will affect these behaviors too. Like if you add to the behaviors too. Like if you add to the behaviors too. Like if you add to the claw MD something like assume you have claw MD something like assume you have claw MD something like assume you have my permission to keep going. If I state my permission to keep going. If I state my permission to keep going. If I state what I want the outcome to be, he'll what I want the outcome to be, he'll what I want the outcome to be, he'll climb till you get there. It's much more climb till you get there. It's much more climb till you get there. It's much more likely to actually get there. They likely to actually get there. They likely to actually get there. They called out here that they were confident called out here that they were confident called out here that they were confident in their guard rails, which meant they in their guard rails, which meant they in their guard rails, which meant they didn't want it to be so reserved and didn't want it to be so reserved and didn't want it to be so reserved and careful. So a lot of what they did, careful. So a lot of what they did, careful. So a lot of what they did, especially early on, was encouraging especially early on, was encouraging especially early on, was encouraging Claude to be bolder. Here's in one of Claude to be bolder. Here's in one of Claude to be bolder. Here's in one of the threads. Claude said, "One small PR the threads. Claude said, "One small PR the threads. Claude said, "One small PR to give code the same timing marks that to give code the same timing marks that to give code the same timing marks that Chad and Co-work already have. I'll put Chad and Co-work already have. I'll put Chad and Co-work already have. I'll put it up this week. Realistically, code's it up this week. Realistically, code's it up this week. Realistically, code's number waits a few days for merge deploy number waits a few days for merge deploy number waits a few days for merge deploy and a baseline window." To which Raymond and a baseline window." To which Raymond and a baseline window." To which Raymond had to respond, "If you put it up right had to respond, "If you put it up right had to respond, "If you put it up right now, I will get it merged and deployed.

now, I will get it merged and deployed. now, I will get it merged and deployed. We have the power to do anything. Please We have the power to do anything. Please We have the power to do anything. Please be braver." Pier will be up within the be braver." Pier will be up within the be braver." Pier will be up within the hour. God, Claude is still so bad at hour. God, Claude is still so bad at hour. God, Claude is still so bad at time estimates. Speaking of which, look time estimates. Speaking of which, look time estimates. Speaking of which, look at that. It actually spun up my thread at that. It actually spun up my thread at that. It actually spun up my thread on my other machine in T3 Code to do the on my other machine in T3 Code to do the on my other machine in T3 Code to do the overhaul of uh ping. Nice. I don't even overhaul of uh ping. Nice. I don't even overhaul of uh ping. Nice. I don't even have anything set up in T3 Code in this have anything set up in T3 Code in this have anything set up in T3 Code in this instance to allow a thread to spawn instance to allow a thread to spawn instance to allow a thread to spawn another thread. I just got it to SSH another thread. I just got it to SSH another thread. I just got it to SSH into another computer, clone the repo, into another computer, clone the repo, into another computer, clone the repo, copy my environment variables, copy the copy my environment variables, copy the copy my environment variables, copy the plan MD file that we had made, and then plan MD file that we had made, and then plan MD file that we had made, and then prompt from one machine to another, spin prompt from one machine to another, spin prompt from one machine to another, spin this up, and then it reappears this up, and then it reappears this up, and then it reappears automatically with the remote automatically with the remote automatically with the remote connections in T3 code. Pretty connections in T3 code. Pretty connections in T3 code. Pretty cool. People aren't prompting boldly cool. People aren't prompting boldly cool. People aren't prompting boldly enough and they're not telling their enough and they're not telling their enough and they're not telling their agents to be bold enough either. And I agents to be bold enough either. And I agents to be bold enough either. And I can already hear the comment section. I can already hear the comment section. I can already hear the comment section. I know how this goes. Well, Theo, some of know how this goes. Well, Theo, some of know how this goes. Well, Theo, some of us work in real code bases where we us work in real code bases where we us work in real code bases where we can't just merge, slop, and break can't just merge, slop, and break can't just merge, slop, and break things. First off, so does anthropic and things. First off, so does anthropic and things. First off, so does anthropic and they break more stuff than I do. Second they break more stuff than I do. Second they break more stuff than I do. Second off, more importantly, the focus of this off, more importantly, the focus of this off, more importantly, the focus of this isn't on merging a bunch of breaking isn't on merging a bunch of breaking isn't on merging a bunch of breaking changes. The point of this is proving changes. The point of this is proving changes. The point of this is proving that you can use the slop to measure that you can use the slop to measure that you can use the slop to measure things to make regressions and problems things to make regressions and problems things to make regressions and problems less likely and also make it easier to less likely and also make it easier to less likely and also make it easier to make improvements. I have a feeling the make improvements. I have a feeling the make improvements. I have a feeling the comment section on this video is going comment section on this video is going comment section on this video is going to literally kill me. So, I'm just going to literally kill me. So, I'm just going to literally kill me. So, I'm just going to move on. I do like this call out that to move on. I do like this call out that to move on. I do like this call out that they put at the end of the section they put at the end of the section they put at the end of the section there. When we started hitting the there. When we started hitting the there. When we started hitting the targets we had set, we noticed the targets we had set, we noticed the targets we had set, we noticed the threads would slow down. Sam went thread threads would slow down. Sam went thread threads would slow down. Sam went thread to thread with the same message. Let's to thread with the same message. Let's to thread with the same message. Let's keep driving this down. The targets are keep driving this down. The targets are keep driving this down. The targets are not the stopping point. What's next? Be not the stopping point. What's next? Be not the stopping point. What's next? Be ambitious. I know y'all think this is

ambitious. I know y'all think this is ambitious. I know y'all think this is cringe, crazy, not useful prompting. cringe, crazy, not useful prompting. cringe, crazy, not useful prompting. What you need to understand is that What you need to understand is that What you need to understand is that these models are effectively word maps these models are effectively word maps these models are effectively word maps and the things it chooses to do because and the things it chooses to do because and the things it chooses to do because the paths in the model drive it there the paths in the model drive it there the paths in the model drive it there when it's just being told fix this thing when it's just being told fix this thing when it's just being told fix this thing and improve this metric are going to be and improve this metric are going to be and improve this metric are going to be the safe happy paths to do that. But as the safe happy paths to do that. But as the safe happy paths to do that. But as soon as you start including words like soon as you start including words like soon as you start including words like ambitious or boil the ocean or push ambitious or boil the ocean or push ambitious or boil the ocean or push beyond what I requested or make weird beyond what I requested or make weird beyond what I requested or make weird and wacky suggestions, you're pulling and wacky suggestions, you're pulling and wacky suggestions, you're pulling the model out of the happy path and the model out of the happy path and the model out of the happy path and towards the sketchier, scarier ones that towards the sketchier, scarier ones that towards the sketchier, scarier ones that is correctly trained to go down less is correctly trained to go down less is correctly trained to go down less because if you don't have the right because if you don't have the right because if you don't have the right tooling, then it could break So, tooling, then it could break So, tooling, then it could break So, if you I know you're saying this if you I know you're saying this if you I know you're saying this ironically, SJO, but I actually think ironically, SJO, but I actually think ironically, SJO, but I actually think this would work. I would unironically this would work. I would unironically this would work. I would unironically guess that if somebody had a model that guess that if somebody had a model that guess that if somebody had a model that they felt like wasn't going far enough, they felt like wasn't going far enough, they felt like wasn't going far enough, if they had a a bot or a hook that would if they had a a bot or a hook that would if they had a a bot or a hook that would automatically append, "Fuck me up, fam," automatically append, "Fuck me up, fam," automatically append, "Fuck me up, fam," to the end of every prompt they send, to the end of every prompt they send, to the end of every prompt they send, they would actually notice a better they would actually notice a better they would actually notice a better experience. It sounds so stupid. It experience. It sounds so stupid. It experience. It sounds so stupid. It really does. And I know you guys are really does. And I know you guys are really does. And I know you guys are going to cook me in the comments for going to cook me in the comments for going to cook me in the comments for this. I don't care. It works.

this. I don't care. It works. this. I don't care. It works. Speaking of all of these fun things that Speaking of all of these fun things that Speaking of all of these fun things that people don't really want to believe in, people don't really want to believe in, people don't really want to believe in, myself included often, the next section myself included often, the next section myself included often, the next section is about taste. Every thread had a named is about taste. Every thread had a named is about taste. Every thread had a named human owner, and Claude highlighted any human owner, and Claude highlighted any human owner, and Claude highlighted any user perceptible change with before and user perceptible change with before and user perceptible change with before and after screenshots or recordings for them after screenshots or recordings for them after screenshots or recordings for them to rule on. Should a table fill in cell to rule on. Should a table fill in cell to rule on. Should a table fill in cell by cell or wait until each row is by cell or wait until each row is by cell or wait until each row is complete? Should a loading skeleton show complete? Should a loading skeleton show complete? Should a loading skeleton show up immediately or only after half a up immediately or only after half a up immediately or only after half a second? Is a wordbyword fade on streamed second? Is a wordbyword fade on streamed second? Is a wordbyword fade on streamed text worth the fifth of a frame budget text worth the fifth of a frame budget text worth the fifth of a frame budget it costs? Claude looked for ways to it costs? Claude looked for ways to it costs? Claude looked for ways to shave milliseconds and we weighed the shave milliseconds and we weighed the shave milliseconds and we weighed the trade-offs. I mean, I can answer all of trade-offs. I mean, I can answer all of trade-offs. I mean, I can answer all of these questions because I know how these questions because I know how these questions because I know how browsers work. Should a table fill in browsers work. Should a table fill in browsers work. Should a table fill in cell by cell or wait until the rows cell by cell or wait until the rows cell by cell or wait until the rows complete? It should wait till rows complete? It should wait till rows complete? It should wait till rows complete. I'd argue it should wait until complete. I'd argue it should wait until complete. I'd argue it should wait until the table's complete. Should a loading the table's complete. Should a loading the table's complete. Should a loading skeleton show up immediately or only skeleton show up immediately or only skeleton show up immediately or only after half a second? You should do a lot after half a second? You should do a lot after half a second? You should do a lot of metrics and analysis to know how long of metrics and analysis to know how long of metrics and analysis to know how long loads take and how wide a range these loads take and how wide a range these loads take and how wide a range these extremes are. Because the worst case extremes are. Because the worst case extremes are. Because the worst case would be that you think the data is would be that you think the data is would be that you think the data is going to take a second. So you show this going to take a second. So you show this going to take a second. So you show this half a second in and then 501 half a second in and then 501 half a second in and then 501 milliseconds in it has the data. So you milliseconds in it has the data. So you milliseconds in it has the data. So you show the skeleton for like two frames at show the skeleton for like two frames at show the skeleton for like two frames at most before you show the real data. You most before you show the real data. You most before you show the real data. You got to tune this one based on real world got to tune this one based on real world got to tune this one based on real world usage. And if you get really fancy with usage. And if you get really fancy with usage. And if you get really fancy with it, you can time other requests and it, you can time other requests and it, you can time other requests and cache some data locally around how long cache some data locally around how long cache some data locally around how long this request usually takes for this user this request usually takes for this user this request usually takes for this user and then make programmatic decisions and then make programmatic decisions and then make programmatic decisions around the skeleton based on that. And around the skeleton based on that. And around the skeleton based on that. And then we have the wordby-word fades for then we have the wordby-word fades for then we have the wordby-word fades for stream text worth a fifth of frame stream text worth a fifth of frame stream text worth a fifth of frame budget. No, wordbyword fade in is almost budget. No, wordbyword fade in is almost budget. No, wordbyword fade in is almost I I would say just straight up isn't I I would say just straight up isn't I I would say just straight up isn't ever worth it. I begrudgingly went from ever worth it. I begrudgingly went from ever worth it. I begrudgingly went from only showing things when the whole only showing things when the whole only showing things when the whole response was done in T3 code to only

response was done in T3 code to only response was done in T3 code to only showing things when the paragraph or new showing things when the paragraph or new showing things when the paragraph or new line is completed or the code block is line is completed or the code block is line is completed or the code block is done. And that allows us to have insane done. And that allows us to have insane done. And that allows us to have insane performance still without any of these performance still without any of these performance still without any of these weird potential issues of like frame weird potential issues of like frame weird potential issues of like frame time being killed by per token stuff. time being killed by per token stuff. time being killed by per token stuff. Yeah. Like I'm happy that they were able Yeah. Like I'm happy that they were able Yeah. Like I'm happy that they were able to work with Claude to find what seemed to work with Claude to find what seemed to work with Claude to find what seemed to be reasonable answers for all of to be reasonable answers for all of to be reasonable answers for all of this, but I now I want to see how this, but I now I want to see how this, but I now I want to see how reasonable these answers are. I'm reasonable these answers are. I'm reasonable these answers are. I'm testing how tables render in Claude. I testing how tables render in Claude. I testing how tables render in Claude. I want you to render some test tables with want you to render some test tables with want you to render some test tables with at least four columns and 10 to 15 rows. at least four columns and 10 to 15 rows. at least four columns and 10 to 15 rows. Can use whatever you want for the data. Can use whatever you want for the data. Can use whatever you want for the data. I just want to see how they render. I just want to see how they render. I just want to see how they render. Let's see how they come in. Let's see how they come in. Let's see how they come in. They're doing They're doing They're doing rowby row it looks like. Uh, nope. That that was pretty brutally Uh, nope. That that was pretty brutally streamed to the side there. Maybe streamed to the side there. Maybe streamed to the side there. Maybe they're doing line or they're doing a they're doing line or they're doing a they're doing line or they're doing a cell by cell. Yeah, that's that's very cell by cell. Yeah, that's that's very cell by cell. Yeah, that's that's very sell by cell. I might need to switch to sell by cell. I might need to switch to sell by cell. I might need to switch to Fable just to make it slower.

This might end up being a bit of a This might end up being a bit of a self-own here if I do this, but uh I self-own here if I do this, but uh I self-own here if I do this, but uh I want to see how this compares to the want to see how this compares to the want to see how this compares to the performance in T3 code for a similar performance in T3 code for a similar performance in T3 code for a similar request. request. request. I prefer that. I don't want to see the I prefer that. I don't want to see the I prefer that. I don't want to see the table while it's still being created. I table while it's still being created. I table while it's still being created. I just want to see it when it's done. To just want to see it when it's done. To just want to see it when it's done. To each their own, but I prefer that. The each their own, but I prefer that. The each their own, but I prefer that. The one thing I don't like, and I'm still one thing I don't like, and I'm still one thing I don't like, and I'm still deciding what I want to do about it, is deciding what I want to do about it, is deciding what I want to do about it, is the title comes in first. and then we the title comes in first. and then we the title comes in first. and then we have to wait for the rest of the have to wait for the rest of the have to wait for the rest of the paragraph. I almost want to hold title paragraph. I almost want to hold title paragraph. I almost want to hold title renders until the content underneath is renders until the content underneath is renders until the content underneath is done. That's like the only change I done. That's like the only change I done. That's like the only change I would make with our behavior here. But would make with our behavior here. But would make with our behavior here. But yeah, you get the idea. These are yeah, you get the idea. These are yeah, you get the idea. These are complex things that you should have complex things that you should have complex things that you should have conversations with your team about and conversations with your team about and conversations with your team about and really make good decisions on. Rowby row really make good decisions on. Rowby row really make good decisions on. Rowby row is what chat's asking about. It's fine, is what chat's asking about. It's fine, is what chat's asking about. It's fine, but I prefer for the table to be done but I prefer for the table to be done but I prefer for the table to be done personally. I will also say that rowby personally. I will also say that rowby personally. I will also say that rowby row renders for like code blocks row renders for like code blocks row renders for like code blocks horrible. Terrible. It breaks all the horrible. Terrible. It breaks all the horrible. Terrible. It breaks all the highlighting. It makes way more render highlighting. It makes way more render highlighting. It makes way more render work than you should need anyways. And I work than you should need anyways. And I work than you should need anyways. And I don't want to read a partial code block. don't want to read a partial code block. don't want to read a partial code block. I just want to read it when it's done. I I just want to read it when it's done. I I just want to read it when it's done. I also find that I don't really sit also find that I don't really sit also find that I don't really sit watching threads anymore. I usually kick watching threads anymore. I usually kick watching threads anymore. I usually kick off the thread and then go somewhere off the thread and then go somewhere off the thread and then go somewhere else and then come back when I have a else and then come back when I have a else and then come back when I have a reason to. So yeah, I now see why they reason to. So yeah, I now see why they reason to. So yeah, I now see why they labeled the section taste because this labeled the section taste because this labeled the section taste because this really is taste. Now we hop to really is taste. Now we hop to really is taste. Now we hop to direction. We kept each thread direction. We kept each thread direction. We kept each thread deliberately narrow, focused on one deliberately narrow, focused on one deliberately narrow, focused on one benchmark or journey, and asked Claude benchmark or journey, and asked Claude benchmark or journey, and asked Claude to find improvements only within that to find improvements only within that to find improvements only within that scope. We thought of threads as 150 scope. We thought of threads as 150 scope. We thought of threads as 150 hammers seeking nails. Most of our calls hammers seeking nails. Most of our calls hammers seeking nails. Most of our calls were about sequencing and user impact, were about sequencing and user impact, were about sequencing and user impact, which surface to prioritize, how to which surface to prioritize, how to which surface to prioritize, how to combine threads that were stepping on combine threads that were stepping on combine threads that were stepping on each other, and when to close a thread

each other, and when to close a thread each other, and when to close a thread that had reached diminishing returns. On that had reached diminishing returns. On that had reached diminishing returns. On one 900 line PR, we got a oneline reply. one 900 line PR, we got a oneline reply. one 900 line PR, we got a oneline reply. Quote, going to gavvel that 2 Quote, going to gavvel that 2 Quote, going to gavvel that 2 millisecond per send is not worth the millisecond per send is not worth the millisecond per send is not worth the complexity of maintaining this build complexity of maintaining this build complexity of maintaining this build plugin. They were adding build plugins plugin. They were adding build plugins plugin. They were adding build plugins to shave 2 milliseconds off message to shave 2 milliseconds off message to shave 2 milliseconds off message sends. This is why you have to be sends. This is why you have to be sends. This is why you have to be careful about how far you let these careful about how far you let these careful about how far you let these things go. Like I I wrote my own V8 things go. Like I I wrote my own V8 things go. Like I I wrote my own V8 based runtime for my cloud, but I had to based runtime for my cloud, but I had to based runtime for my cloud, but I had to like push for that. We're nearing the like push for that. We're nearing the like push for that. We're nearing the point where the agents will autonomously point where the agents will autonomously point where the agents will autonomously do like that. So make sure you do like that. So make sure you do like that. So make sure you look out for unnecessarily huge changes look out for unnecessarily huge changes look out for unnecessarily huge changes for unimportantly small wins. This for unimportantly small wins. This for unimportantly small wins. This sounds like an interesting section. An 8 sounds like an interesting section. An 8 sounds like an interesting section. An 8 millisecond budget. One of our side millisecond budget. One of our side millisecond budget. One of our side quests shows everything working together quests shows everything working together quests shows everything working together to demonstrate an optimization to a reax to demonstrate an optimization to a reax to demonstrate an optimization to a reax used in the live syntax highlighting. used in the live syntax highlighting. used in the live syntax highlighting. Claude attached a screen recording of a Claude attached a screen recording of a Claude attached a screen recording of a longer answer streaming in the lab. In longer answer streaming in the lab. In longer answer streaming in the lab. In the corner, it added a frame rate the corner, it added a frame rate the corner, it added a frame rate readout computed in the page from readout computed in the page from readout computed in the page from animation frame timestamps. Well, yeah, animation frame timestamps. Well, yeah, animation frame timestamps. Well, yeah, for those who don't know, 8 milliseconds for those who don't know, 8 milliseconds for those who don't know, 8 milliseconds is how long you get per frame at 120 Hz. is how long you get per frame at 120 Hz. is how long you get per frame at 120 Hz. A lot of people have 120 Hz displays, A lot of people have 120 Hz displays, A lot of people have 120 Hz displays, whether it's their iPhone, their whether it's their iPhone, their whether it's their iPhone, their MacBook, or any gaming monitor, and even MacBook, or any gaming monitor, and even MacBook, or any gaming monitor, and even some modern TVs. It's a good target to some modern TVs. It's a good target to some modern TVs. It's a good target to aim for because somebody with a very aim for because somebody with a very aim for because somebody with a very expensive pro MacBook is going to want expensive pro MacBook is going to want expensive pro MacBook is going to want the page to run at the 120 FPS. This is the page to run at the 120 FPS. This is the page to run at the 120 FPS. This is actually kind of a sick bench. Are we actually kind of a sick bench. Are we actually kind of a sick bench. Are we capped at 60 fps? Can we try to drive capped at 60 fps? Can we try to drive capped at 60 fps? Can we try to drive scroll and stream smoothness up to 120?

scroll and stream smoothness up to 120? scroll and stream smoothness up to 120? If I understand correctly, your rig may If I understand correctly, your rig may If I understand correctly, your rig may not support this. Right. Today's rig not support this. Right. Today's rig not support this. Right. Today's rig ticks at 60 Hz because headless chromium ticks at 60 Hz because headless chromium ticks at 60 Hz because headless chromium does by default. I believe it can be does by default. I believe it can be does by default. I believe it can be driven at 120. Confirming that first, driven at 120. Confirming that first, driven at 120. Confirming that first, then I'll rerun the eval at an 8.3 then I'll rerun the eval at an 8.3 then I'll rerun the eval at an 8.3 millisecond frame budget. Update on the millisecond frame budget. Update on the millisecond frame budget. Update on the 120 Hz rig. It works. Deterministic 120 120 Hz rig. It works. Deterministic 120 120 Hz rig. It works. Deterministic 120 Hz frame stepping in headless Chrome via Hz frame stepping in headless Chrome via Hz frame stepping in headless Chrome via dev tools. Begin frame control. Exactly dev tools. Begin frame control. Exactly dev tools. Begin frame control. Exactly 240 frames with 240 begin frames at 833 240 frames with 240 begin frames at 833 240 frames with 240 begin frames at 833 mms. This is this is not Opus 55. This mms. This is this is not Opus 55. This mms. This is this is not Opus 55. This is Claude slop. This is hardcore Claude is Claude slop. This is hardcore Claude is Claude slop. This is hardcore Claude slop. Then Raymond amazing cook. Yeah, slop. Then Raymond amazing cook. Yeah, slop. Then Raymond amazing cook. Yeah, I love this. I also am just now I love this. I also am just now I love this. I also am just now realizing one of the things that I realizing one of the things that I realizing one of the things that I really like about quad tag. Notice what really like about quad tag. Notice what really like about quad tag. Notice what isn't in this thread. Notice that we isn't in this thread. Notice that we isn't in this thread. Notice that we don't have code. Notice that we don't don't have code. Notice that we don't don't have code. Notice that we don't have tool calls and the responses for have tool calls and the responses for have tool calls and the responses for those tool calls. Notice that it's not those tool calls. Notice that it's not those tool calls. Notice that it's not listing all the commands it ran. Notice listing all the commands it ran. Notice listing all the commands it ran. Notice it's not showing all of the thinking it's not showing all of the thinking it's not showing all of the thinking traces for the sections of thinking that traces for the sections of thinking that traces for the sections of thinking that it did. I like this because they don't it did. I like this because they don't it did. I like this because they don't really matter anymore. I'm considering really matter anymore. I'm considering really matter anymore. I'm considering adding a feature to T3 code that just adding a feature to T3 code that just adding a feature to T3 code that just outright hides them or maybe gives you a outright hides them or maybe gives you a outright hides them or maybe gives you a small ticker like I've run this many small ticker like I've run this many small ticker like I've run this many tools so far because I don't care. I tools so far because I don't care. I tools so far because I don't care. I just want to know when it's done. I also just want to know when it's done. I also just want to know when it's done. I also can't help but note the irony that a lot can't help but note the irony that a lot can't help but note the irony that a lot of the performance things they're trying of the performance things they're trying of the performance things they're trying to fix here are not in Slack. They're in to fix here are not in Slack. They're in to fix here are not in Slack. They're in the Claude web app and desktop app, the Claude web app and desktop app, the Claude web app and desktop app, which they are notably not using for any which they are notably not using for any which they are notably not using for any of this. First off, that's funny because of this. First off, that's funny because of this. First off, that's funny because Slack's a better experience with Claude

Slack's a better experience with Claude Slack's a better experience with Claude than Claude code. But more importantly, than Claude code. But more importantly, than Claude code. But more importantly, it's funny because a lot of these it's funny because a lot of these it's funny because a lot of these performance issues are things that they performance issues are things that they performance issues are things that they show in the desktop app and on cloud.ai show in the desktop app and on cloud.ai show in the desktop app and on cloud.ai that they do not show in Slack. Just that they do not show in Slack. Just that they do not show in Slack. Just saying. Once the mechanisms and ambition saying. Once the mechanisms and ambition saying. Once the mechanisms and ambition were established, Claude got to work. were established, Claude got to work. were established, Claude got to work. Each painted frame had a budget of 8.3 Each painted frame had a budget of 8.3 Each painted frame had a budget of 8.3 milliseconds. So Claude stepped through milliseconds. So Claude stepped through milliseconds. So Claude stepped through a long reply frame by frame, timing each a long reply frame by frame, timing each a long reply frame by frame, timing each one to find the slow parts. It one to find the slow parts. It one to find the slow parts. It eliminated some work that was O to the eliminated some work that was O to the eliminated some work that was O to the message length, so however many things message length, so however many things message length, so however many things are in the message, which is a lot of are in the message, which is a lot of are in the message, which is a lot of work it was doing per chunk. By work it was doing per chunk. By work it was doing per chunk. By memorizing finished blocks, moving memorizing finished blocks, moving memorizing finished blocks, moving tokenization logic from growing code tokenization logic from growing code tokenization logic from growing code fences to a worker, and revealed tables fences to a worker, and revealed tables fences to a worker, and revealed tables cell by cell. Oh, that's cool. Uh we cell by cell. Oh, that's cool. Uh we cell by cell. Oh, that's cool. Uh we were doing perch chunk memoization for were doing perch chunk memoization for were doing perch chunk memoization for T3 chat in February of last year. It's T3 chat in February of last year. It's T3 chat in February of last year. It's been about 22 months or so. The moved been about 22 months or so. The moved been about 22 months or so. The moved tokenization logic from growing code tokenization logic from growing code tokenization logic from growing code fences to a worker. This is the early fences to a worker. This is the early fences to a worker. This is the early pieces of what I was suggesting earlier pieces of what I was suggesting earlier pieces of what I was suggesting earlier which is using workers and wom to run which is using workers and wom to run which is using workers and wom to run your syntax highlighting and code block your syntax highlighting and code block your syntax highlighting and code block stuff so that it's not blocking the main stuff so that it's not blocking the main stuff so that it's not blocking the main thread. Also, you can cache the result thread. Also, you can cache the result thread. Also, you can cache the result of that in memoized chunks on the of that in memoized chunks on the of that in memoized chunks on the outside of the like wom side and the outside of the like wom side and the outside of the like wom side and the worker side in the react world. So, you worker side in the react world. So, you worker side in the react world. So, you don't have to re-trigger the WASM on don't have to re-trigger the WASM on don't have to re-trigger the WASM on every change. You can also reveal the every change. You can also reveal the every change. You can also reveal the table cell by cell. Yeah. So, I guess I table cell by cell. Yeah. So, I guess I table cell by cell. Yeah. So, I guess I had the answer to my question earlier had the answer to my question earlier had the answer to my question earlier about how they're doing the table stuff about how they're doing the table stuff about how they're doing the table stuff here. Just need to scroll more. In that here. Just need to scroll more. In that here. Just need to scroll more. In that one thread, we landed nearly 60 PRs.

one thread, we landed nearly 60 PRs. one thread, we landed nearly 60 PRs. This is just the thread about the 8.33 This is just the thread about the 8.33 This is just the thread about the 8.33 millisecond render times. Long replies millisecond render times. Long replies millisecond render times. Long replies were blocking the main thread for 200 were blocking the main thread for 200 were blocking the main thread for 200 milliseconds in total where they used to milliseconds in total where they used to milliseconds in total where they used to block for over 750. Ran on about a third block for over 750. Ran on about a third block for over 750. Ran on about a third as much CPU and it held 120 FPS from as much CPU and it held 120 FPS from as much CPU and it held 120 FPS from start to finish on 120 Hz MacBook. The start to finish on 120 Hz MacBook. The start to finish on 120 Hz MacBook. The 120 Hz rig itself became a nightly job 120 Hz rig itself became a nightly job 120 Hz rig itself became a nightly job with Claude watching for regressions. with Claude watching for regressions. with Claude watching for regressions. Awesome. We love that. Oh, I hadn't Awesome. We love that. Oh, I hadn't Awesome. We love that. Oh, I hadn't thought of that. Agore, good point. We thought of that. Agore, good point. We thought of that. Agore, good point. We could just use Jev to do the syntax could just use Jev to do the syntax could just use Jev to do the syntax highlighting. That would solve all these highlighting. That would solve all these highlighting. That would solve all these problems. Oh, that's actually really problems. Oh, that's actually really problems. Oh, that's actually really funny. They cite the tweet that I I funny. They cite the tweet that I I funny. They cite the tweet that I I actually people saw what my post about actually people saw what my post about actually people saw what my post about this as dunking. It was not meant to be this as dunking. It was not meant to be this as dunking. It was not meant to be a dunk at all. I was actually trying to a dunk at all. I was actually trying to a dunk at all. I was actually trying to start a conversation around whether or start a conversation around whether or start a conversation around whether or not streaming matters. This is mine my not streaming matters. This is mine my not streaming matters. This is mine my my quote tweet here. I don't want my quote tweet here. I don't want my quote tweet here. I don't want responses streamed anymore. Just give me responses streamed anymore. Just give me responses streamed anymore. Just give me the whole paragraph or block when it's the whole paragraph or block when it's the whole paragraph or block when it's done. To be very clear, I'm referring to done. To be very clear, I'm referring to done. To be very clear, I'm referring to token by token streaming. You should token by token streaming. You should token by token streaming. You should stream down paragraphs, code blocks, stream down paragraphs, code blocks, stream down paragraphs, code blocks, tool calls, etc. Just stop sending me tool calls, etc. Just stop sending me tool calls, etc. Just stop sending me incomplete text. I also called out that incomplete text. I also called out that incomplete text. I also called out that I feel obligated to mention this I feel obligated to mention this I feel obligated to mention this performance work is very hard and I'm performance work is very hard and I'm performance work is very hard and I'm impressed about the work they did here.

impressed about the work they did here. impressed about the work they did here. Particularly funny that I have this when Particularly funny that I have this when Particularly funny that I have this when in the end it was all just claude in the end it was all just claude in the end it was all just claude running in a loop. Anyways, when they running in a loop. Anyways, when they running in a loop. Anyways, when they started this performance sprint, they started this performance sprint, they started this performance sprint, they hadn't planned to hill climb on the hadn't planned to hill climb on the hadn't planned to hill climb on the milliseconds between frames while milliseconds between frames while milliseconds between frames while streaming, but it turned out that they streaming, but it turned out that they streaming, but it turned out that they could count them. So, it being countable could count them. So, it being countable could count them. So, it being countable meant Claude could climb it. So, what's meant Claude could climb it. So, what's meant Claude could climb it. So, what's next? Cloud AI and the desktop app are next? Cloud AI and the desktop app are next? Cloud AI and the desktop app are now three times faster than they were in now three times faster than they were in now three times faster than they were in early August. and the ratchets should early August. and the ratchets should early August. and the ratchets should keep them there. But they're not done. keep them there. But they're not done. keep them there. But they're not done. The 95th percentile other journeys and The 95th percentile other journeys and The 95th percentile other journeys and very long conversations still have room very long conversations still have room very long conversations still have room to improve. In a separate post, we'll to improve. In a separate post, we'll to improve. In a separate post, we'll also write about some of the side quests also write about some of the side quests also write about some of the side quests that took them upstream during the that took them upstream during the that took them upstream during the sprint with contributions landing in sprint with contributions landing in sprint with contributions landing in Electron, Chromium, Node, and more. Electron, Chromium, Node, and more. Electron, Chromium, Node, and more. They'll share the results internally. They'll share the results internally. They'll share the results internally. Isaac put it best. You cannot have Isaac put it best. You cannot have Isaac put it best. You cannot have convinced me that this was possible even convinced me that this was possible even convinced me that this was possible even 6 months ago. Yep, this would have been 6 months ago. Yep, this would have been 6 months ago. Yep, this would have been very hard to convince me 6 months ago. very hard to convince me 6 months ago. very hard to convince me 6 months ago. We expect to keep working this way one We expect to keep working this way one We expect to keep working this way one thread at a time at any scale. The thread at a time at any scale. The thread at a time at any scale. The channel's still going. I'll shout out channel's still going. I'll shout out channel's still going. I'll shout out Anthony. I know I've been harsh towards Anthony. I know I've been harsh towards Anthony. I know I've been harsh towards Claude Desktop, but I know he's been Claude Desktop, but I know he's been Claude Desktop, but I know he's been working really hard on improving it. I working really hard on improving it. I working really hard on improving it. I did actually use it a bunch when I was did actually use it a bunch when I was did actually use it a bunch when I was testing Opus 55 and it has made a lot of testing Opus 55 and it has made a lot of testing Opus 55 and it has made a lot of improvements. So, shout out to him and improvements. So, shout out to him and improvements. So, shout out to him and the rest of the team making Claude Code the rest of the team making Claude Code the rest of the team making Claude Code Desktop way better. Anony's awesome. And Desktop way better. Anony's awesome. And Desktop way better. Anony's awesome. And also, I do love that Boris is also, I do love that Boris is also, I do love that Boris is encouraging them to be more ambitious.

encouraging them to be more ambitious. encouraging them to be more ambitious. That is a very good thing for him to do That is a very good thing for him to do That is a very good thing for him to do as the leader. This was a genuinely as the leader. This was a genuinely as the leader. This was a genuinely really fun read. I enjoyed this a lot. I really fun read. I enjoyed this a lot. I really fun read. I enjoyed this a lot. I didn't learn quite as much as I hoped didn't learn quite as much as I hoped didn't learn quite as much as I hoped to, but it did really emphasize a lot of to, but it did really emphasize a lot of to, but it did really emphasize a lot of the things I've already been trying to the things I've already been trying to the things I've already been trying to do. giving the model the ability to do. giving the model the ability to do. giving the model the ability to measure things so that it can fix them measure things so that it can fix them measure things so that it can fix them rather than hoping it will figure that rather than hoping it will figure that rather than hoping it will figure that all out when you ask it to fix things. all out when you ask it to fix things. all out when you ask it to fix things. If you tell the model improve If you tell the model improve If you tell the model improve performance, it will do its best based performance, it will do its best based performance, it will do its best based on its theories to improve the on its theories to improve the on its theories to improve the performance. But if you start by telling performance. But if you start by telling performance. But if you start by telling it to find ways to measure performance, it to find ways to measure performance, it to find ways to measure performance, validate those findings and then tell it validate those findings and then tell it validate those findings and then tell it to go improve the performance, it'll to go improve the performance, it'll to go improve the performance, it'll improve a lot. This is really cool and I improve a lot. This is really cool and I improve a lot. This is really cool and I think we can all learn a lot around how think we can all learn a lot around how think we can all learn a lot around how they are prompting, how they're thinking they are prompting, how they're thinking they are prompting, how they're thinking about these changes and how they are about these changes and how they are about these changes and how they are validating the work that they're doing validating the work that they're doing validating the work that they're doing because that is going to be more and because that is going to be more and because that is going to be more and more important as things continue to more important as things continue to more important as things continue to change. If you still think that reading change. If you still think that reading change. If you still think that reading the code is the solution to all these the code is the solution to all these the code is the solution to all these problems, then you're going to have a problems, then you're going to have a problems, then you're going to have a whole separate set of problems and a whole separate set of problems and a whole separate set of problems and a whole lot less solutions. Figuring out whole lot less solutions. Figuring out whole lot less solutions. Figuring out how to work with and around the slop how to work with and around the slop how to work with and around the slop really kind of is becoming the meta. I really kind of is becoming the meta. I really kind of is becoming the meta. I know a lot of engineers are going to know a lot of engineers are going to know a lot of engineers are going to hate that. I personally find it kind of hate that. I personally find it kind of hate that. I personally find it kind of fun. It's a new type of engineering that fun. It's a new type of engineering that fun. It's a new type of engineering that requires new types of cleverness in requires new types of cleverness in requires new types of cleverness in design to set up systems and structures design to set up systems and structures design to set up systems and structures that allow your agents to operate that allow your agents to operate that allow your agents to operate effectively and efficiently. I'm effectively and efficiently. I'm effectively and efficiently. I'm thankful I'm finding more opportunities thankful I'm finding more opportunities thankful I'm finding more opportunities to deep dive in these fun engineering to deep dive in these fun engineering to deep dive in these fun engineering topics again. I really am enjoying it.

topics again. I really am enjoying it. topics again. I really am enjoying it. Thank you all for watching this video as Thank you all for watching this video as Thank you all for watching this video as well as all the kind words on the recent well as all the kind words on the recent well as all the kind words on the recent React Native one. It's such a relief to React Native one. It's such a relief to React Native one. It's such a relief to be able to dive into the engineering be able to dive into the engineering be able to dive into the engineering topics again where it makes sense. Let topics again where it makes sense. Let topics again where it makes sense. Let me know how you feel about all of this me know how you feel about all of this me know how you feel about all of this and what you learned in the comments. and what you learned in the comments. and what you learned in the comments. And until next time, peace nerds.