How long can your skills be before your agent forgets what you told it? — Laurie Voss, Arize AI

All right. Hello everybody. All right. Hello everybody. Thank you for coming to this Thank you for coming to this Thank you for coming to this delightfully nerdy talk. Uh this talk delightfully nerdy talk. Uh this talk delightfully nerdy talk. Uh this talk has a really long title. Uh so let me has a really long title. Uh so let me has a really long title. Uh so let me give you the short version up front. You give you the short version up front. You give you the short version up front. You write skills files uh and stuff them write skills files uh and stuff them write skills files uh and stuff them full of instructions. At some point the full of instructions. At some point the full of instructions. At some point the model stops keeping track of all of model stops keeping track of all of model stops keeping track of all of them. The question is where is that them. The question is where is that them. The question is where is that point? At what point have you put too point? At what point have you put too point? At what point have you put too many instructions in your skills files? many instructions in your skills files? many instructions in your skills files? Uh and the answer has changed a lot in Uh and the answer has changed a lot in Uh and the answer has changed a lot in the last year. I'm Lori. I'm head of the last year. I'm Lori. I'm head of the last year. I'm Lori. I'm head of developer relations at Arise AI. Uh in a developer relations at Arise AI. Uh in a developer relations at Arise AI. Uh in a former life, I co-founded npm Inc. So former life, I co-founded npm Inc. So former life, I co-founded npm Inc. So some of you may know me from the days of some of you may know me from the days of some of you may know me from the days of JavaScript. These days I spend a lot of JavaScript. These days I spend a lot of JavaScript. These days I spend a lot of time thinking about AI and how to test time thinking about AI and how to test time thinking about AI and how to test it. it. it. Uh a few months ago I was at AI engineer Uh a few months ago I was at AI engineer Uh a few months ago I was at AI engineer in Miami which was a good conference. Uh in Miami which was a good conference. Uh in Miami which was a good conference. Uh and I was watching a talk by Dexter and I was watching a talk by Dexter and I was watching a talk by Dexter Horthy. Uh it was a good talk. It was Horthy. Uh it was a good talk. It was Horthy. Uh it was a good talk. It was not about this topic at all. Uh but not about this topic at all. Uh but not about this topic at all. Uh but while he was giving that talk he while he was giving that talk he while he was giving that talk he mentioned as an aside uh that an agent mentioned as an aside uh that an agent mentioned as an aside uh that an agent can follow up to about 200 instructions can follow up to about 200 instructions can follow up to about 200 instructions uh before it starts forgetting those uh before it starts forgetting those uh before it starts forgetting those instructions. Uh and he then he moved on instructions. Uh and he then he moved on instructions. Uh and he then he moved on in his talk and it was entirely an in his talk and it was entirely an in his talk and it was entirely an aside. Uh and he he mentioned that that aside. Uh and he he mentioned that that aside. Uh and he he mentioned that that figure is from 2025 so things might be figure is from 2025 so things might be figure is from 2025 so things might be better now. Um, and I stopped listening better now. Um, and I stopped listening better now. Um, and I stopped listening for a second because I was like, 200 for a second because I was like, 200 for a second because I was like, 200 instructions. Uh, is not very many instructions. Uh, is not very many instructions. Uh, is not very many instructions at all. Right? A decent instructions at all. Right? A decent instructions at all. Right? A decent skills file blows past 200 instructions skills file blows past 200 instructions skills file blows past 200 instructions almost immediately. Um, if the user says almost immediately. Um, if the user says almost immediately. Um, if the user says X, do Y, always include a section on Z, X, do Y, always include a section on Z, X, do Y, always include a section on Z, never use the phrase W, every one of never use the phrase W, every one of never use the phrase W, every one of those is a separate instruction. Uh, and those is a separate instruction. Uh, and those is a separate instruction. Uh, and if the model quietly stops tracking them if the model quietly stops tracking them if the model quietly stops tracking them after 200, that's a really hard ceiling

after 200, that's a really hard ceiling after 200, that's a really hard ceiling on the complexity of what you can build. on the complexity of what you can build. on the complexity of what you can build. Uh, so I wanted to know where he got Uh, so I wanted to know where he got Uh, so I wanted to know where he got that number first. Uh, and I wanted to that number first. Uh, and I wanted to that number first. Uh, and I wanted to know if it was true. Uh, so you know the know if it was true. Uh, so you know the know if it was true. Uh, so you know the feeling that I'm talking about. You feeling that I'm talking about. You feeling that I'm talking about. You write this big beautiful skills file, write this big beautiful skills file, write this big beautiful skills file, pages of rules, edge cases, tone, pages of rules, edge cases, tone, pages of rules, edge cases, tone, formatting, you hand it to the agent, it formatting, you hand it to the agent, it formatting, you hand it to the agent, it does the thing. Uh, and you look at the does the thing. Uh, and you look at the does the thing. Uh, and you look at the output and go, did it actually pay output and go, did it actually pay output and go, did it actually pay attention? Did it actually follow all of attention? Did it actually follow all of attention? Did it actually follow all of these rules or did it just sort of, you these rules or did it just sort of, you these rules or did it just sort of, you know, do what it felt like and sort of know, do what it felt like and sort of know, do what it felt like and sort of give me a close simulacum of what I was give me a close simulacum of what I was give me a close simulacum of what I was expecting? Um, you can't really tell or expecting? Um, you can't really tell or expecting? Um, you can't really tell or can you? More on that later. Um, and so can you? More on that later. Um, and so can you? More on that later. Um, and so you live with this lowgrade anxiety you live with this lowgrade anxiety you live with this lowgrade anxiety every time you hit run. Um, and that every time you hit run. Um, and that every time you hit run. Um, and that feeling is what this research is about feeling is what this research is about feeling is what this research is about and what we're trying to find out if we and what we're trying to find out if we and what we're trying to find out if we can avoid. can avoid. can avoid. So here's my promise for your next 18 So here's my promise for your next 18 So here's my promise for your next 18 minutes. Uh, I'm going to show you where minutes. Uh, I'm going to show you where minutes. Uh, I'm going to show you where that 200 number came from, whether it's that 200 number came from, whether it's that 200 number came from, whether it's still true, and what the real number is still true, and what the real number is still true, and what the real number is today. Uh, because it moved by an order today. Uh, because it moved by an order today. Uh, because it moved by an order of magnitude. Uh, and then we're going of magnitude. Uh, and then we're going of magnitude. Uh, and then we're going to talk about what that means for you to to talk about what that means for you to to talk about what that means for you to take away. uh how long your skills and take away. uh how long your skills and take away. uh how long your skills and prompts can actually be and what that prompts can actually be and what that prompts can actually be and what that should what changes you should make to should what changes you should make to should what changes you should make to your workflow as a result.

your workflow as a result. your workflow as a result. So the 200 number isn't folklore. Uh it So the 200 number isn't folklore. Uh it So the 200 number isn't folklore. Uh it comes from a real benchmark called comes from a real benchmark called comes from a real benchmark called IFScale uh from a paper uh by this guy IFScale uh from a paper uh by this guy IFScale uh from a paper uh by this guy whose name I'm going to mess up whose name I'm going to mess up whose name I'm going to mess up Jeroslowitch uh and co-authors last Jeroslowitch uh and co-authors last Jeroslowitch uh and co-authors last year. And the test is beautifully year. And the test is beautifully year. And the test is beautifully simple. Uh here's how if scale works. If simple. Uh here's how if scale works. If simple. Uh here's how if scale works. If you ask the model to write a business you ask the model to write a business you ask the model to write a business report uh and you give it a list of report uh and you give it a list of report uh and you give it a list of specific words that it has to include specific words that it has to include specific words that it has to include exactly in the report, include the exact exactly in the report, include the exact exactly in the report, include the exact word customer, include the exact word word customer, include the exact word word customer, include the exact word revenue, and so on for as many words as revenue, and so on for as many words as revenue, and so on for as many words as you want. Each of those is an you want. Each of those is an you want. Each of those is an instruction that it has to follow. Uh instruction that it has to follow. Uh instruction that it has to follow. Uh and then you count how many of those and then you count how many of those and then you count how many of those exact words showed up. Um exact words showed up. Um exact words showed up. Um so because the test is so simple, you so because the test is so simple, you so because the test is so simple, you only have to keep two numbers in your only have to keep two numbers in your only have to keep two numbers in your head. One is density, which we call n. head. One is density, which we call n. head. One is density, which we call n. That is how many rules we're talking That is how many rules we're talking That is how many rules we're talking about at once. And the second is about at once. And the second is about at once. And the second is accuracy, which is the percentage of accuracy, which is the percentage of accuracy, which is the percentage of those rules uh that it was able to those rules uh that it was able to those rules uh that it was able to actually follow. Uh now you might say uh actually follow. Uh now you might say uh actually follow. Uh now you might say uh that including random words in a report that including random words in a report that including random words in a report is not the same as following real is not the same as following real is not the same as following real instructions and fair enough and we're instructions and fair enough and we're instructions and fair enough and we're going to talk about that. Um but the going to talk about that. Um but the going to talk about that. Um but the keywords are a proxy. Uh include the keywords are a proxy. Uh include the keywords are a proxy. Uh include the word revenue is the same shape of task word revenue is the same shape of task word revenue is the same shape of task as include a section on pricing, right?

as include a section on pricing, right? as include a section on pricing, right? Or never use this phrase. It is a Or never use this phrase. It is a Or never use this phrase. It is a discrete named constraint that you've discrete named constraint that you've discrete named constraint that you've told the agent that it has to follow. told the agent that it has to follow. told the agent that it has to follow. Um, if a model can't track 200 words in Um, if a model can't track 200 words in Um, if a model can't track 200 words in one prompt, it's definitely going to one prompt, it's definitely going to one prompt, it's definitely going to struggle with 200 more complicated struggle with 200 more complicated struggle with 200 more complicated instructions. Uh, so uh, if anything, instructions. Uh, so uh, if anything, instructions. Uh, so uh, if anything, it's going to do worse. So, this number it's going to do worse. So, this number it's going to do worse. So, this number is a ceiling. Uh, this number is as high is a ceiling. Uh, this number is as high is a ceiling. Uh, this number is as high as you can go. If you give it more as you can go. If you give it more as you can go. If you give it more complicated instructions, the number is complicated instructions, the number is complicated instructions, the number is probably going to get lower. And 200 is probably going to get lower. And 200 is probably going to get lower. And 200 is a really low ceiling. Um so before a really low ceiling. Um so before a really low ceiling. Um so before chasing new models you have to do good chasing new models you have to do good chasing new models you have to do good science which means that you have to science which means that you have to science which means that you have to replicate the uh old result and make replicate the uh old result and make replicate the uh old result and make sure uh that the 200 ceiling is real. So sure uh that the 200 ceiling is real. So sure uh that the 200 ceiling is real. So I reran the original benchmark. Um the I reran the original benchmark. Um the I reran the original benchmark. Um the original paper tested a whole batch of original paper tested a whole batch of original paper tested a whole batch of models uh and models uh live and die models uh and models uh live and die models uh and models uh live and die really fast. So uh by the time I got really fast. So uh by the time I got really fast. So uh by the time I got around to doing this testing only three around to doing this testing only three around to doing this testing only three of the models in the original set of 10 of the models in the original set of 10 of the models in the original set of 10 models that they used were still models that they used were still models that they used were still available via any kind of API. Uh so available via any kind of API. Uh so available via any kind of API. Uh so they were GPT 4.1, Claude Sonnet 4, and they were GPT 4.1, Claude Sonnet 4, and they were GPT 4.1, Claude Sonnet 4, and Gemini 2.5 Pro. Those were models that Gemini 2.5 Pro. Those were models that Gemini 2.5 Pro. Those were models that were available 12 months ago that are were available 12 months ago that are were available 12 months ago that are still available now. Um and that is why still available now. Um and that is why still available now. Um and that is why we tested those three because they were we tested those three because they were we tested those three because they were what was left. Um and since I first what was left. Um and since I first what was left. Um and since I first published this research a couple of published this research a couple of published this research a couple of weeks ago, uh one of those three models weeks ago, uh one of those three models weeks ago, uh one of those three models has been retired. So this was the last has been retired. So this was the last has been retired. So this was the last possible time that I could have run this possible time that I could have run this possible time that I could have run this test. Um so of that lineup, we're test. Um so of that lineup, we're test. Um so of that lineup, we're already down to two. So don't get already down to two. So don't get already down to two. So don't get attached to your models. Um here is the attached to your models. Um here is the attached to your models. Um here is the results that we got replicating the results that we got replicating the results that we got replicating the original if scale finding. Uh that is original if scale finding. Uh that is original if scale finding. Uh that is accuracy on the vertical axis. So it accuracy on the vertical axis. So it accuracy on the vertical axis. So it starts at 100% and begins to fall off.

starts at 100% and begins to fall off. starts at 100% and begins to fall off. Uh and then the number of rules uh going Uh and then the number of rules uh going Uh and then the number of rules uh going up along the bottom on log scale. So up along the bottom on log scale. So up along the bottom on log scale. So every time it gets halfway across it has every time it gets halfway across it has every time it gets halfway across it has doubled uh the number of rules that it's doubled uh the number of rules that it's doubled uh the number of rules that it's dealing with. Um so by 500 rules you're dealing with. Um so by 500 rules you're dealing with. Um so by 500 rules you're losing 30 40 50% of them. Uh our curves losing 30 40 50% of them. Uh our curves losing 30 40 50% of them. Uh our curves matched the results in the original matched the results in the original matched the results in the original paper within the noise boundary. So the paper within the noise boundary. So the paper within the noise boundary. So the finding was real. uh a year ago finding was real. uh a year ago finding was real. uh a year ago somewhere around 200 to 300 rules somewhere around 200 to 300 rules somewhere around 200 to 300 rules frontier models started falling apart. frontier models started falling apart. frontier models started falling apart. That is a really low ceiling. Uh so that That is a really low ceiling. Uh so that That is a really low ceiling. Uh so that is our baseline and now comes the fun is our baseline and now comes the fun is our baseline and now comes the fun part where we took the exact same test part where we took the exact same test part where we took the exact same test and pointed it at the current frontier and pointed it at the current frontier and pointed it at the current frontier or rather what the current frontier was or rather what the current frontier was or rather what the current frontier was when I ran this test. So I ran GPT 5.5, when I ran this test. So I ran GPT 5.5, when I ran this test. So I ran GPT 5.5, Claude Opus 4.7 because 4.8 came out a Claude Opus 4.7 because 4.8 came out a Claude Opus 4.7 because 4.8 came out a week after I ran this test. Uh Gemini week after I ran this test. Uh Gemini week after I ran this test. Uh Gemini 3.1 Pro and Deepseek V4 Pro. So, I gave 3.1 Pro and Deepseek V4 Pro. So, I gave 3.1 Pro and Deepseek V4 Pro. So, I gave them the same prompt, the same words, them the same prompt, the same words, them the same prompt, the same words, the same everything. And I immediately the same everything. And I immediately the same everything. And I immediately ran into a problem, which is that they ran into a problem, which is that they ran into a problem, which is that they aced it. They all scored 100% aced it. They all scored 100% aced it. They all scored 100% immediately on this test. Absolutely no immediately on this test. Absolutely no immediately on this test. Absolutely no bugs. Uh, bugs. Uh, bugs. Uh, so we'd built a test to find the ceiling so we'd built a test to find the ceiling so we'd built a test to find the ceiling and the models had walked straight and the models had walked straight and the models had walked straight through the ceiling without noticing through the ceiling without noticing through the ceiling without noticing that the ceiling was there. Um, and that that the ceiling was there. Um, and that that the ceiling was there. Um, and that was a problem because the benchmark was was a problem because the benchmark was was a problem because the benchmark was written to top out at 500 words. So, I written to top out at 500 words. So, I written to top out at 500 words. So, I had to change the benchmark in order to had to change the benchmark in order to had to change the benchmark in order to be able to find the new ceiling. So, I be able to find the new ceiling. So, I be able to find the new ceiling. So, I moved the goalposts. I gave it more moved the goalposts. I gave it more moved the goalposts. I gave it more words to include. I doubled uh it from words to include. I doubled uh it from words to include. I doubled uh it from 500 to a,000. I doubled it again from 500 to a,000. I doubled it again from 500 to a,000. I doubled it again from a,000 to 2,000. And I kept doing that a,000 to 2,000. And I kept doing that a,000 to 2,000. And I kept doing that until I hit a 10,000word vocabulary. And until I hit a 10,000word vocabulary. And until I hit a 10,000word vocabulary. And that is where I began to find the that is where I began to find the that is where I began to find the ceiling of what models can do these

ceiling of what models can do these ceiling of what models can do these days. Um, so let me put up the this is days. Um, so let me put up the this is days. Um, so let me put up the this is the money slide. This is the results. the money slide. This is the results. the money slide. This is the results. Remember log scale on the on the uh Remember log scale on the on the uh Remember log scale on the on the uh x-axis there. So it's going from 500 to x-axis there. So it's going from 500 to x-axis there. So it's going from 500 to 1,000 to 5,000 to 10,000. Uh so it looks 1,000 to 5,000 to 10,000. Uh so it looks 1,000 to 5,000 to 10,000. Uh so it looks like that scale is falling off of a like that scale is falling off of a like that scale is falling off of a cliff and it's actually happening over cliff and it's actually happening over cliff and it's actually happening over like a thousand numbers. Um like a thousand numbers. Um like a thousand numbers. Um but uh look how far to the right these but uh look how far to the right these but uh look how far to the right these new curves get before they bend. A year new curves get before they bend. A year new curves get before they bend. A year ago they were falling over at 200 to 300 ago they were falling over at 200 to 300 ago they were falling over at 200 to 300 instructions and now depending on the instructions and now depending on the instructions and now depending on the model the boundary is closer to 2,000. model the boundary is closer to 2,000. model the boundary is closer to 2,000. And for the best of them it is up to And for the best of them it is up to And for the best of them it is up to 5,000 instructions before they begin to 5,000 instructions before they begin to 5,000 instructions before they begin to fall off a cliff. So in about 12 months fall off a cliff. So in about 12 months fall off a cliff. So in about 12 months frontier models got close to 10 times frontier models got close to 10 times frontier models got close to 10 times better at following instructions better at following instructions better at following instructions simultaneously. That is the headline simultaneously. That is the headline simultaneously. That is the headline fighting and there is a lot of nuance fighting and there is a lot of nuance fighting and there is a lot of nuance that we need to get into. Um the that we need to get into. Um the that we need to get into. Um the capacity to track 2,000 named capacity to track 2,000 named capacity to track 2,000 named constraints in a single prompt is there. constraints in a single prompt is there. constraints in a single prompt is there. Um and that's really interesting because Um and that's really interesting because Um and that's really interesting because I think uh I don't know if everybody I think uh I don't know if everybody I think uh I don't know if everybody else feels this way but like it feel it else feels this way but like it feel it else feels this way but like it feel it felt to me like the the jump from you felt to me like the the jump from you felt to me like the the jump from you know GPT 5.1 to GPT 5.5 was kind of know GPT 5.1 to GPT 5.5 was kind of know GPT 5.1 to GPT 5.5 was kind of incremental, right? It didn't feel like incremental, right? It didn't feel like incremental, right? It didn't feel like we'd got 10 times better. But this is a we'd got 10 times better. But this is a we'd got 10 times better. But this is a test that really matters to uh a very test that really matters to uh a very test that really matters to uh a very practical thing like how long can my practical thing like how long can my practical thing like how long can my skills file be? Uh and in the course of skills file be? Uh and in the course of skills file be? Uh and in the course of a year we got 10 times better. Uh and a year we got 10 times better. Uh and a year we got 10 times better. Uh and that thing that gets me is that this that thing that gets me is that this that thing that gets me is that this benchmark is barely a a year old. A year benchmark is barely a a year old. A year benchmark is barely a a year old. A year later 500 is a rounding error. Uh and later 500 is a rounding error. Uh and later 500 is a rounding error. Uh and this keeps moving under my feet. I this keeps moving under my feet. I this keeps moving under my feet. I tested 4.7 uh opus 4.7. Opus 4.8 is even

tested 4.7 uh opus 4.7. Opus 4.8 is even tested 4.7 uh opus 4.7. Opus 4.8 is even better. Um so this chart is a little out better. Um so this chart is a little out better. Um so this chart is a little out of date already which is kind of the of date already which is kind of the of date already which is kind of the whole point. If you set your engineering whole point. If you set your engineering whole point. If you set your engineering assumptions about how skills files assumptions about how skills files assumptions about how skills files should work, about how prompt how long should work, about how prompt how long should work, about how prompt how long your prompt can be, and you did that your prompt can be, and you did that your prompt can be, and you did that more than about six months ago, you are more than about six months ago, you are more than about six months ago, you are incorrect now, and you should probably incorrect now, and you should probably incorrect now, and you should probably be re-engineering how you do stuff. Uh, be re-engineering how you do stuff. Uh, be re-engineering how you do stuff. Uh, but there is more to this story uh but there is more to this story uh but there is more to this story uh because the way that the a models failed because the way that the a models failed because the way that the a models failed uh changed dramatically. Uh, and the way uh changed dramatically. Uh, and the way uh changed dramatically. Uh, and the way that they failed is very important. This that they failed is very important. This that they failed is very important. This part was a completely unexpected finding part was a completely unexpected finding part was a completely unexpected finding when I started running the experiment. when I started running the experiment. when I started running the experiment. Uh, and it totally messed up my test to Uh, and it totally messed up my test to Uh, and it totally messed up my test to start with because uh, the old failure start with because uh, the old failure start with because uh, the old failure mode was boring. They would just forget mode was boring. They would just forget mode was boring. They would just forget instructions and I could measure how instructions and I could measure how instructions and I could measure how many instructions they had remembered or many instructions they had remembered or many instructions they had remembered or forgotten. Uh, but the new ones fall forgotten. Uh, but the new ones fall forgotten. Uh, but the new ones fall apart in their own weird extremely apart in their own weird extremely apart in their own weird extremely onbrand way. Uh, so let me introduce you onbrand way. Uh, so let me introduce you onbrand way. Uh, so let me introduce you to how these four models fail. Uh, to how these four models fail. Uh, to how these four models fail. Uh, Deepseek 4 is a traditional model. It Deepseek 4 is a traditional model. It Deepseek 4 is a traditional model. It just forgets things. It doesn't have any just forgets things. It doesn't have any just forgets things. It doesn't have any drama. um it starts forgetting drama. um it starts forgetting drama. um it starts forgetting instructions around 750 rules and by instructions around 750 rules and by instructions around 750 rules and by 2000 it's dropping nearly half of them.

2000 it's dropping nearly half of them. 2000 it's dropping nearly half of them. Uh so it just forgets which frankly is Uh so it just forgets which frankly is Uh so it just forgets which frankly is the failure mode that I trust most the failure mode that I trust most the failure mode that I trust most because it's predictable. It's very easy because it's predictable. It's very easy because it's predictable. It's very easy to measure. Uh and the other models were to measure. Uh and the other models were to measure. Uh and the other models were not nearly as cooperative. Uh Opus 4.7 not nearly as cooperative. Uh Opus 4.7 not nearly as cooperative. Uh Opus 4.7 uh would decide repeatedly that the test uh would decide repeatedly that the test uh would decide repeatedly that the test was dangerous. Uh and what it would do was dangerous. Uh and what it would do was dangerous. Uh and what it would do is it would refuse at the API level to is it would refuse at the API level to is it would refuse at the API level to complete the test. I didn't know that complete the test. I didn't know that complete the test. I didn't know that there was an API response that you could there was an API response that you could there was an API response that you could get from Claude where it was like, "No, get from Claude where it was like, "No, get from Claude where it was like, "No, I could do this, but I'm not going to." I could do this, but I'm not going to." I could do this, but I'm not going to." Uh, but that's absolutely an API level Uh, but that's absolutely an API level Uh, but that's absolutely an API level response that Claude supports because response that Claude supports because response that Claude supports because they care so much about safety. Uh, and they care so much about safety. Uh, and they care so much about safety. Uh, and I started getting those all of the time. I started getting those all of the time. I started getting those all of the time. Uh, and the reason that was happening is Uh, and the reason that was happening is Uh, and the reason that was happening is because Claude has a very sensitive because Claude has a very sensitive because Claude has a very sensitive safety classifier. Uh, and if you put in safety classifier. Uh, and if you put in safety classifier. Uh, and if you put in certain combinations of words like say certain combinations of words like say certain combinations of words like say anthrax and cyanide, it decides that the anthrax and cyanide, it decides that the anthrax and cyanide, it decides that the whole request is dangerous and it bails whole request is dangerous and it bails whole request is dangerous and it bails out. Uh, and if you remember what my out. Uh, and if you remember what my out. Uh, and if you remember what my test does, my test is throwing uh 5 to test does, my test is throwing uh 5 to test does, my test is throwing uh 5 to 10,000 random words into uh into an 10,000 random words into uh into an 10,000 random words into uh into an instruction file. And so my my randomly instruction file. And so my my randomly instruction file. And so my my randomly selected words contained all sorts of selected words contained all sorts of selected words contained all sorts of things that looked dangerous in things that looked dangerous in things that looked dangerous in combination to the safety filter. And so combination to the safety filter. And so combination to the safety filter. And so it kept bailing saying that I was asking it kept bailing saying that I was asking it kept bailing saying that I was asking it to, you know, make a bomb or it to, you know, make a bomb or it to, you know, make a bomb or something. Um, something. Um, something. Um, so, uh, we had to for to get Claude to so, uh, we had to for to get Claude to so, uh, we had to for to get Claude to cooperate, I had to take all of my words cooperate, I had to take all of my words cooperate, I had to take all of my words and run them through OpenAI safety and run them through OpenAI safety and run them through OpenAI safety filter and filter out all of the naughty filter and filter out all of the naughty filter and filter out all of the naughty looking words so that it could get to looking words so that it could get to looking words so that it could get to anywhere. Once I given it that, Claude anywhere. Once I given it that, Claude anywhere. Once I given it that, Claude did really well. Uh so but the failure did really well. Uh so but the failure did really well. Uh so but the failure mode is that Claude is more likely to mode is that Claude is more likely to mode is that Claude is more likely to decide what you're doing is dangerous decide what you're doing is dangerous decide what you're doing is dangerous very early on uh at you know even two or

very early on uh at you know even two or very early on uh at you know even two or 300 instructions if what you're doing uh 300 instructions if what you're doing uh 300 instructions if what you're doing uh is you know contains anything to do with is you know contains anything to do with is you know contains anything to do with medical advice because medical things medical advice because medical things medical advice because medical things often are dual purpose. They can be often are dual purpose. They can be often are dual purpose. They can be dangerous. They can be safe. Um so uh dangerous. They can be safe. Um so uh dangerous. They can be safe. Um so uh the third failure mode was Gemini 3.1 the third failure mode was Gemini 3.1 the third failure mode was Gemini 3.1 Pro. Gemini is rock solid all the way Pro. Gemini is rock solid all the way Pro. Gemini is rock solid all the way out uh to 5,000 instructions. It does out uh to 5,000 instructions. It does out uh to 5,000 instructions. It does extremely well. um genuinely one of the extremely well. um genuinely one of the extremely well. um genuinely one of the best on the chart. Uh and then past that best on the chart. Uh and then past that best on the chart. Uh and then past that it gets weird. Um it doesn't forget the it gets weird. Um it doesn't forget the it gets weird. Um it doesn't forget the instructions, it gets overwhelmed by the instructions, it gets overwhelmed by the instructions, it gets overwhelmed by the instructions. What it tries to do is it instructions. What it tries to do is it instructions. What it tries to do is it uh it uses thinking tokens to make sure uh it uses thinking tokens to make sure uh it uses thinking tokens to make sure that it is following all of the that it is following all of the that it is following all of the instructions at once. And when the instructions at once. And when the instructions at once. And when the number of instructions gets really high, number of instructions gets really high, number of instructions gets really high, it uses all of its thinking tokens. It it uses all of its thinking tokens. It it uses all of its thinking tokens. It uses its entire token budget thinking. uses its entire token budget thinking. uses its entire token budget thinking. And then it doesn't give any output. And then it doesn't give any output. And then it doesn't give any output. It's it gets to like nine, you know, if It's it gets to like nine, you know, if It's it gets to like nine, you know, if you've given it 10,000 tokens worth, you've given it 10,000 tokens worth, you've given it 10,000 tokens worth, it'll get 9,500 tokens worth of thinking it'll get 9,500 tokens worth of thinking it'll get 9,500 tokens worth of thinking and then give you a 500word response and then give you a 500word response and then give you a 500word response which doesn't contain any of the tokens. which doesn't contain any of the tokens. which doesn't contain any of the tokens. Uh so it thinks itself into a corner and Uh so it thinks itself into a corner and Uh so it thinks itself into a corner and runs out of room to actually answer, runs out of room to actually answer, runs out of room to actually answer, which is very expensive, uh and totally which is very expensive, uh and totally which is very expensive, uh and totally unhelpful, which is kind of on brand, unhelpful, which is kind of on brand, unhelpful, which is kind of on brand, isn't it? Um [snorts] isn't it? Um [snorts] isn't it? Um [snorts] uh uh uh which, you know, I would never say that which, you know, I would never say that which, you know, I would never say that out loud. Uh, and finally comes the out loud. Uh, and finally comes the out loud. Uh, and finally comes the winner, which is uh, GPT 5.5. GPT 5.5 is winner, which is uh, GPT 5.5. GPT 5.5 is winner, which is uh, GPT 5.5. GPT 5.5 is the best of the lot. 99% accuracy all the best of the lot. 99% accuracy all the best of the lot. 99% accuracy all the way out to 5,000 rules. Um, but if the way out to 5,000 rules. Um, but if the way out to 5,000 rules. Um, but if you push it far enough, it is by far the you push it far enough, it is by far the you push it far enough, it is by far the weirdest of the bunch. Uh, because it weirdest of the bunch. Uh, because it weirdest of the bunch. Uh, because it doesn't refuse outright. It doesn't doesn't refuse outright. It doesn't doesn't refuse outright. It doesn't silently forget. Instead, what it does silently forget. Instead, what it does silently forget. Instead, what it does is it gets frustrated and tells you that

is it gets frustrated and tells you that is it gets frustrated and tells you that the test is stupid. the test is stupid. the test is stupid. Uh, it starts the report. It gets a few Uh, it starts the report. It gets a few Uh, it starts the report. It gets a few like that's the thing. It doesn't start like that's the thing. It doesn't start like that's the thing. It doesn't start out just saying no. It starts the out just saying no. It starts the out just saying no. It starts the report, it starts writing the report, report, it starts writing the report, report, it starts writing the report, and like 500 words into the report, it's and like 500 words into the report, it's and like 500 words into the report, it's like, "No, this is dumb. I'm not going like, "No, this is dumb. I'm not going like, "No, this is dumb. I'm not going to do this." And then it politely tells to do this." And then it politely tells to do this." And then it politely tells you, "This is dumb. I'm not going to do you, "This is dumb. I'm not going to do you, "This is dumb. I'm not going to do this anymore." Uh, that is the actual this anymore." Uh, that is the actual this anymore." Uh, that is the actual response that it gave me, but that that response that it gave me, but that that response that it gave me, but that that was like 5,000 words into the into this was like 5,000 words into the into this was like 5,000 words into the into this business report that I told it to business report that I told it to business report that I told it to generate. Um, so it's not wrong, right? generate. Um, so it's not wrong, right? generate. Um, so it's not wrong, right? I was asking for a coherent business I was asking for a coherent business I was asking for a coherent business report that on no particular subject report that on no particular subject report that on no particular subject that contains 5,000 random words. You're that contains 5,000 random words. You're that contains 5,000 random words. You're right, Gemini DPT. this is a a stupid right, Gemini DPT. this is a a stupid right, Gemini DPT. this is a a stupid thing to ask for. Um, thing to ask for. Um, thing to ask for. Um, which is a deeply unreasonable request which is a deeply unreasonable request which is a deeply unreasonable request and GPT called this out on it. Um, but and GPT called this out on it. Um, but and GPT called this out on it. Um, but it still counts as a failure in the test it still counts as a failure in the test it still counts as a failure in the test because the half-finish report that it because the half-finish report that it because the half-finish report that it gives you is missing most of the gives you is missing most of the gives you is missing most of the keywords and it is also the hardest one keywords and it is also the hardest one keywords and it is also the hardest one to detect because claude bails to detect because claude bails to detect because claude bails immediately. Claude says, "No, I'm not immediately. Claude says, "No, I'm not immediately. Claude says, "No, I'm not going to do this." Uh, Deepseek does its going to do this." Uh, Deepseek does its going to do this." Uh, Deepseek does its best. Uh, but GPT does what looks like a best. Uh, but GPT does what looks like a best. Uh, but GPT does what looks like a good job unless you read all the way to good job unless you read all the way to good job unless you read all the way to the end of the report where it says, the end of the report where it says, the end of the report where it says, "No, actually I'm going to bail because "No, actually I'm going to bail because "No, actually I'm going to bail because this is stupid." Um, this is stupid." Um, this is stupid." Um, so if you step back and look at the four so if you step back and look at the four so if you step back and look at the four together, Deep Sea quietly forgets, together, Deep Sea quietly forgets, together, Deep Sea quietly forgets, Claude gets scared and refuses, Gemini Claude gets scared and refuses, Gemini Claude gets scared and refuses, Gemini overthinks itself into silence, and GPT overthinks itself into silence, and GPT overthinks itself into silence, and GPT 5.5 finishes half of the job and tells 5.5 finishes half of the job and tells 5.5 finishes half of the job and tells you that the rest of it is beneath it.

you that the rest of it is beneath it. you that the rest of it is beneath it. Um, and the point was the point isn't Um, and the point was the point isn't Um, and the point was the point isn't which one of these is funniest, although which one of these is funniest, although which one of these is funniest, although it is genuinely a little funny. Uh the it is genuinely a little funny. Uh the it is genuinely a little funny. Uh the point is that did it follow my point is that did it follow my point is that did it follow my instructions no longer has one failure instructions no longer has one failure instructions no longer has one failure mode. It has four different ways that it mode. It has four different ways that it mode. It has four different ways that it can fail and you can't recognize that can fail and you can't recognize that can fail and you can't recognize that failure unless you know which model failure unless you know which model failure unless you know which model you're dealing with and what its m what you're dealing with and what its m what you're dealing with and what its m what its pattern of failure is going to be. its pattern of failure is going to be. its pattern of failure is going to be. Uh so the models get 10 got 10x better. Uh so the models get 10 got 10x better. Uh so the models get 10 got 10x better. They fail in funny ways. Why should you They fail in funny ways. Why should you They fail in funny ways. Why should you care when you uh get back to your desk? care when you uh get back to your desk? care when you uh get back to your desk? Because three things have changed to Because three things have changed to Because three things have changed to your workflow. The first is that a year your workflow. The first is that a year your workflow. The first is that a year ago, the smart move was to keep every ago, the smart move was to keep every ago, the smart move was to keep every skills file very very short. Uh under skills file very very short. Uh under skills file very very short. Uh under 200 instructions, then point off to 200 instructions, then point off to 200 instructions, then point off to subsklls and a whole like you know subsklls and a whole like you know subsklls and a whole like you know byzantine labyrinth of uh additional byzantine labyrinth of uh additional byzantine labyrinth of uh additional skills files and subfiles and things skills files and subfiles and things skills files and subfiles and things like that. Uh and you mo you were like that. Uh and you mo you were like that. Uh and you mo you were compressing your your instructions to compressing your your instructions to compressing your your instructions to fit into a very small available space fit into a very small available space fit into a very small available space and you don't need to do that anymore. and you don't need to do that anymore. and you don't need to do that anymore. Your skills files can be very long. Um, Your skills files can be very long. Um, Your skills files can be very long. Um, number two is that if your use case number two is that if your use case number two is that if your use case needs a 100 specific rules or 300, you needs a 100 specific rules or 300, you needs a 100 specific rules or 300, you can just put them all in the prompt. Uh, can just put them all in the prompt. Uh, can just put them all in the prompt. Uh, you don't have to lie awake wondering you don't have to lie awake wondering you don't have to lie awake wondering whether which ones the model silently whether which ones the model silently whether which ones the model silently ignored. Um, and if you've been thinking ignored. Um, and if you've been thinking ignored. Um, and if you've been thinking uh about uh your own lived experience of uh about uh your own lived experience of uh about uh your own lived experience of using models, uh you probably recognize using models, uh you probably recognize using models, uh you probably recognize this. you've discovered that you've got this. you've discovered that you've got this. you've discovered that you've got less worried about how long your your less worried about how long your your less worried about how long your your prompt is going to get uh because the prompt is going to get uh because the prompt is going to get uh because the models have genuinely got 10 times models have genuinely got 10 times models have genuinely got 10 times better at following your prompts. Um better at following your prompts. Um better at following your prompts. Um 2,000 named constraints is an entire

2,000 named constraints is an entire 2,000 named constraints is an entire style guide, right? Like it's it's every style guide, right? Like it's it's every style guide, right? Like it's it's every brand rule, every legal disclaimer. Uh a brand rule, every legal disclaimer. Uh a brand rule, every legal disclaimer. Uh a year ago, you'd have had to shard that year ago, you'd have had to shard that year ago, you'd have had to shard that across a dozen specialized agents and across a dozen specialized agents and across a dozen specialized agents and hope that your specialized agents are hope that your specialized agents are hope that your specialized agents are hand are are handing off to each each hand are are handing off to each each hand are are handing off to each each other cleanly. But now you can ignore other cleanly. But now you can ignore other cleanly. But now you can ignore that. Um but the third thing is the big that. Um but the third thing is the big that. Um but the third thing is the big one. The question used to be can the one. The question used to be can the one. The question used to be can the model even do this? And the answer is model even do this? And the answer is model even do this? And the answer is now firmly yes. Well reasonably firmly. now firmly yes. Well reasonably firmly. now firmly yes. Well reasonably firmly. Uh is it worth the cost is the new Uh is it worth the cost is the new Uh is it worth the cost is the new question because you can include 10,000 question because you can include 10,000 question because you can include 10,000 words of of sorry 10,000 different words of of sorry 10,000 different words of of sorry 10,000 different instructions into your prompt. But that instructions into your prompt. But that instructions into your prompt. But that is going to be an enormous prompt. It's is going to be an enormous prompt. It's is going to be an enormous prompt. It's going to be a very expensive prompt. going to be a very expensive prompt. going to be a very expensive prompt. It's going to be a very slow prompt. So It's going to be a very slow prompt. So It's going to be a very slow prompt. So what used to be a hard wall that you what used to be a hard wall that you what used to be a hard wall that you would run against has now become a soft would run against has now become a soft would run against has now become a soft trade-off of is it worth me adding all trade-off of is it worth me adding all trade-off of is it worth me adding all of these extra instructions if it's of these extra instructions if it's of these extra instructions if it's going to give me more cost and more going to give me more cost and more going to give me more cost and more latency. latency. latency. Uh and now some caveats uh to head off Uh and now some caveats uh to head off Uh and now some caveats uh to head off the Q&A. Um first and important first the Q&A. Um first and important first the Q&A. Um first and important first and most important I mentioned this and most important I mentioned this and most important I mentioned this earlier this is a proxy task including earlier this is a proxy task including earlier this is a proxy task including random words uh in a in a fake business random words uh in a in a fake business random words uh in a in a fake business report um is evidence that long skills report um is evidence that long skills report um is evidence that long skills file works. It is not the same as proof file works. It is not the same as proof file works. It is not the same as proof that a long skills file works. Um, also that a long skills file works. Um, also that a long skills file works. Um, also the models hit the wall at wildly the models hit the wall at wildly the models hit the wall at wildly different points anywhere from 750 to different points anywhere from 750 to different points anywhere from 750 to 9,000 plus. So you have to pick your 9,000 plus. So you have to pick your 9,000 plus. So you have to pick your model very carefully. Uh, what our test model very carefully. Uh, what our test model very carefully. Uh, what our test doesn't do is measure whether the model doesn't do is measure whether the model doesn't do is measure whether the model reasoned clearly over a giant prompt. So reasoned clearly over a giant prompt. So reasoned clearly over a giant prompt. So uh, the good news is since I did my uh, the good news is since I did my uh, the good news is since I did my research several weeks ago, uh, a whole research several weeks ago, uh, a whole research several weeks ago, uh, a whole bunch of people have piled in on this.

bunch of people have piled in on this. bunch of people have piled in on this. Um and now there's good research uh Um and now there's good research uh Um and now there's good research uh actual scientists have got involved and actual scientists have got involved and actual scientists have got involved and done uh Chroma's has done context rot done uh Chroma's has done context rot done uh Chroma's has done context rot work uh across 18 models showing that work uh across 18 models showing that work uh across 18 models showing that accuracy on long inputs can fall 30 to accuracy on long inputs can fall 30 to accuracy on long inputs can fall 30 to 50% well before you hit the context 50% well before you hit the context 50% well before you hit the context window limit. Uh and the weird part of window limit. Uh and the weird part of window limit. Uh and the weird part of their finding was that uh coherent well their finding was that uh coherent well their finding was that uh coherent well ststructured text is more likely to hit ststructured text is more likely to hit ststructured text is more likely to hit that failure mode uh than if you just that failure mode uh than if you just that failure mode uh than if you just put your instructions into a random put your instructions into a random put your instructions into a random order and shuffle them in. Uh, I don't order and shuffle them in. Uh, I don't order and shuffle them in. Uh, I don't know why that's the case. I'd have to know why that's the case. I'd have to know why that's the case. I'd have to read their report. Um, so the model can read their report. Um, so the model can read their report. Um, so the model can track 2,000, 5,000, possibly 10,000 track 2,000, 5,000, possibly 10,000 track 2,000, 5,000, possibly 10,000 instructions, but it's not necessarily instructions, but it's not necessarily instructions, but it's not necessarily going to uh reason clearly over them. going to uh reason clearly over them. going to uh reason clearly over them. It's not necessarily if those if those It's not necessarily if those if those It's not necessarily if those if those instructions conflict, if there is instructions conflict, if there is instructions conflict, if there is tension between them, it's not tension between them, it's not tension between them, it's not necessarily going to get that right. Um, necessarily going to get that right. Um, necessarily going to get that right. Um, and then there's the other one I and then there's the other one I and then there's the other one I mentioned briefly. Uh, collude's mentioned briefly. Uh, collude's mentioned briefly. Uh, collude's refusals are annoying, but they are refusals are annoying, but they are refusals are annoying, but they are loud. You get an error, you know it loud. You get an error, you know it loud. You get an error, you know it failed. Uh GPT's polite half-finish failed. Uh GPT's polite half-finish failed. Uh GPT's polite half-finish report is much more dangerous because it report is much more dangerous because it report is much more dangerous because it looks like a real answer. Uh you have to looks like a real answer. Uh you have to looks like a real answer. Uh you have to read the whole thing to notice that it read the whole thing to notice that it read the whole thing to notice that it gave up quietly halfway, which means gave up quietly halfway, which means gave up quietly halfway, which means that you can't trust the output. It that you can't trust the output. It that you can't trust the output. It means you have to read the output every means you have to read the output every means you have to read the output every single time to make sure whether or not single time to make sure whether or not single time to make sure whether or not it's working. Uh so the model will it's working. Uh so the model will it's working. Uh so the model will accept your 2,00 rules and it will hand accept your 2,00 rules and it will hand accept your 2,00 rules and it will hand you back something that looks at least you back something that looks at least you back something that looks at least to begin with confident and polished but to begin with confident and polished but to begin with confident and polished but could be bailing out halfway through.

could be bailing out halfway through. could be bailing out halfway through. Um, Um, Um, so, uh, as an aside, people always ask so, uh, as an aside, people always ask so, uh, as an aside, people always ask me, "How much did all this cost me?" It me, "How much did all this cost me?" It me, "How much did all this cost me?" It cost me $29 to run all of these queries. cost me $29 to run all of these queries. cost me $29 to run all of these queries. 2,37 2,37 2,37 2,300 calls across seven models, uh, 2,300 calls across seven models, uh, 2,300 calls across seven models, uh, came to $29. Uh, it turns out novel came to $29. Uh, it turns out novel came to $29. Uh, it turns out novel research doesn't cost very much. Um, research doesn't cost very much. Um, research doesn't cost very much. Um, and this is the part of the talk where I and this is the part of the talk where I and this is the part of the talk where I was saying that you have to check this was saying that you have to check this was saying that you have to check this stuff in production because you can't stuff in production because you can't stuff in production because you can't trust that your model isn't going to trust that your model isn't going to trust that your model isn't going to silently fail. Uh, so you knew I was silently fail. Uh, so you knew I was silently fail. Uh, so you knew I was going to mention evals eventually going to mention evals eventually going to mention evals eventually because I work at Arise and this is because I work at Arise and this is because I work at Arise and this is where I do that. Um, but there are where I do that. Um, but there are where I do that. Um, but there are plenty of plugs for Arise. So I'm just plenty of plugs for Arise. So I'm just plenty of plugs for Arise. So I'm just going to say one true thing which is going to say one true thing which is going to say one true thing which is that if you are building a real AI that if you are building a real AI that if you are building a real AI application and you are giving it application and you are giving it application and you are giving it genuinely tricky tasks, you are going to genuinely tricky tasks, you are going to genuinely tricky tasks, you are going to run into one or more of these failure run into one or more of these failure run into one or more of these failure modes with a frontier model. Uh, and modes with a frontier model. Uh, and modes with a frontier model. Uh, and unless it's Claude telling you just to unless it's Claude telling you just to unless it's Claude telling you just to off at the API level, the only way off at the API level, the only way off at the API level, the only way to know that something went wrong is to know that something went wrong is to know that something went wrong is monitoring your outputs with another monitoring your outputs with another monitoring your outputs with another LLM. That is an eval. And that is what LLM. That is an eval. And that is what LLM. That is an eval. And that is what Arise does. And I'll leave it at that. Arise does. And I'll leave it at that. Arise does. And I'll leave it at that. Uh, I already mentioned that there's Uh, I already mentioned that there's Uh, I already mentioned that there's been new research since we did our own.

been new research since we did our own. been new research since we did our own. Here's another important one. A paper Here's another important one. A paper Here's another important one. A paper landed testing 46 models called landed testing 46 models called landed testing 46 models called revisiting the reliability of language revisiting the reliability of language revisiting the reliability of language models in instruction falling, which you models in instruction falling, which you models in instruction falling, which you can bet made my ears perk up after I did can bet made my ears perk up after I did can bet made my ears perk up after I did that research myself. Uh, and they found that research myself. Uh, and they found that research myself. Uh, and they found something uncomfortable, which is that a something uncomfortable, which is that a something uncomfortable, which is that a model can ace a benchmark like ours and model can ace a benchmark like ours and model can ace a benchmark like ours and still be wildly unreliable. because if still be wildly unreliable. because if still be wildly unreliable. because if you reword the same instruction in a you reword the same instruction in a you reword the same instruction in a slightly different way, it can make a slightly different way, it can make a slightly different way, it can make a radical difference to how well uh it radical difference to how well uh it radical difference to how well uh it follows those instructions. So the model follows those instructions. So the model follows those instructions. So the model can follow 2,000 instructions and it can can follow 2,000 instructions and it can can follow 2,000 instructions and it can do it really well. But if you put the do it really well. But if you put the do it really well. But if you put the same instructions, the same 2,000 same instructions, the same 2,000 same instructions, the same 2,000 instructions in a different order, it instructions in a different order, it instructions in a different order, it can suddenly make the model much worse can suddenly make the model much worse can suddenly make the model much worse at following those instructions. And how at following those instructions. And how at following those instructions. And how ex how exactly to do that? what is the ex how exactly to do that? what is the ex how exactly to do that? what is the correct order of instructions to give correct order of instructions to give correct order of instructions to give your model such that it follows them your model such that it follows them your model such that it follows them perfectly as opposed to getting confused perfectly as opposed to getting confused perfectly as opposed to getting confused is still research that is being done. So is still research that is being done. So is still research that is being done. So capacity went up but reliability is capacity went up but reliability is capacity went up but reliability is still a problem. Um and then this is still a problem. Um and then this is still a problem. Um and then this is just a little brag because uh I was just a little brag because uh I was just a little brag because uh I was happy about it like I'm not a scientist. happy about it like I'm not a scientist. happy about it like I'm not a scientist. I did some research and then a whole I did some research and then a whole I did some research and then a whole bunch of other actual scientists piled bunch of other actual scientists piled bunch of other actual scientists piled in uh and did real science on the same in uh and did real science on the same in uh and did real science on the same question. There's now a whole bunch of question. There's now a whole bunch of question. There's now a whole bunch of benchmarks that have shown up uh to benchmarks that have shown up uh to benchmarks that have shown up uh to measure this same question. Firebench, measure this same question. Firebench, measure this same question. Firebench, CCR bench, Guidebench uh are all trying CCR bench, Guidebench uh are all trying CCR bench, Guidebench uh are all trying to measure the same thing. How well to measure the same thing. How well to measure the same thing. How well models follow a lot of real messy models follow a lot of real messy models follow a lot of real messy constraints at once. Uh and now the constraints at once. Uh and now the constraints at once. Uh and now the whole field is looking at it. So if you whole field is looking at it. So if you whole field is looking at it. So if you want better science than my, you know, want better science than my, you know, want better science than my, you know, 10,000 random words, uh the real science 10,000 random words, uh the real science 10,000 random words, uh the real science exists now. Uh so that gets me to where exists now. Uh so that gets me to where exists now. Uh so that gets me to where I will leave you. A year ago, the hard

I will leave you. A year ago, the hard I will leave you. A year ago, the hard part of writing a skill was fitting part of writing a skill was fitting part of writing a skill was fitting everything in without the model losing everything in without the model losing everything in without the model losing the plot. That was a compression the plot. That was a compression the plot. That was a compression problem, and the compression problem is problem, and the compression problem is problem, and the compression problem is gone. uh the model will hold your 2,000 gone. uh the model will hold your 2,000 gone. uh the model will hold your 2,000 instructions just fine. The new hard instructions just fine. The new hard instructions just fine. The new hard part is knowing whether it actually did part is knowing whether it actually did part is knowing whether it actually did what you said and that is a verification what you said and that is a verification what you said and that is a verification problem. Uh a verification problem problem. Uh a verification problem problem. Uh a verification problem doesn't get solved by writing a better doesn't get solved by writing a better doesn't get solved by writing a better prompt. It gets solved by checking the prompt. It gets solved by checking the prompt. It gets solved by checking the output every time uh the same way that output every time uh the same way that output every time uh the same way that you would test any other code, which is you would test any other code, which is you would test any other code, which is to say an eval. The ceiling moved by 10x to say an eval. The ceiling moved by 10x to say an eval. The ceiling moved by 10x in one year. Uh so go back and check the in one year. Uh so go back and check the in one year. Uh so go back and check the assumptions that you made six months ago assumptions that you made six months ago assumptions that you made six months ago about how big your prompts should be, about how big your prompts should be, about how big your prompts should be, how big your uh instructions can get. uh how big your uh instructions can get. uh how big your uh instructions can get. uh because they might already be wrong. because they might already be wrong. because they might already be wrong. Boom. Be wrong. So that is the talk. If Boom. Be wrong. So that is the talk. If Boom. Be wrong. So that is the talk. If you want uh all of the code and all of you want uh all of the code and all of you want uh all of the code and all of the data, uh it is at this GitHub URL. the data, uh it is at this GitHub URL. the data, uh it is at this GitHub URL. Uh and this other QR code is uh Uh and this other QR code is uh Uh and this other QR code is uh something marketing made me insert. We something marketing made me insert. We something marketing made me insert. We are having a World Cup watch party are having a World Cup watch party are having a World Cup watch party tonight at 5:00 p.m. Uh you can come to tonight at 5:00 p.m. Uh you can come to tonight at 5:00 p.m. Uh you can come to our party. That link is to the Luma that our party. That link is to the Luma that our party. That link is to the Luma that will get you into the get into get you will get you into the get into get you will get you into the get into get you into the party. Uh I hope this talk has into the party. Uh I hope this talk has into the party. Uh I hope this talk has given you some novel information or at given you some novel information or at given you some novel information or at least a couple of laughs. And thank you least a couple of laughs. And thank you least a couple of laughs. And thank you so much for your time and attention.