Computer Use at the Edge of the Statistical Precipice — Pierluca D'Oro, Programma Labs

Hi everyone, I am Pugadoro and I'm the Hi everyone, I am Pugadoro and I'm the founder at Programmabs and today I'm founder at Programmabs and today I'm founder at Programmabs and today I'm going to talk about computer use agency going to talk about computer use agency going to talk about computer use agency evaluation and most of the work um and evaluation and most of the work um and evaluation and most of the work um and the details about it are in a in a paper the details about it are in a in a paper the details about it are in a in a paper with this title and I did this work with this title and I did this work with this title and I did this work while Meta super intelligent labs with while Meta super intelligent labs with while Meta super intelligent labs with the collaborators you see on this slide. the collaborators you see on this slide. the collaborators you see on this slide. And so to start I want to introduce this And so to start I want to introduce this And so to start I want to introduce this type of uh agent is a weird type of type of uh agent is a weird type of type of uh agent is a weird type of agent that I call a replay agent. So agent that I call a replay agent. So agent that I call a replay agent. So imagine we run this process um we run imagine we run this process um we run imagine we run this process um we run our frontier a frontier model a good one our frontier a frontier model a good one our frontier a frontier model a good one uh on a benchmark we like and then for uh on a benchmark we like and then for uh on a benchmark we like and then for every task we collect uh a successful every task we collect uh a successful every task we collect uh a successful trace or a successful trajectory and we trace or a successful trajectory and we trace or a successful trajectory and we have a recorded tape of this type. So have a recorded tape of this type. So have a recorded tape of this type. So the actions might be tapping uh typing the actions might be tapping uh typing the actions might be tapping uh typing scrolling and we record this and then we scrolling and we record this and then we scrolling and we record this and then we do this for all the tasks in the do this for all the tasks in the do this for all the tasks in the benchmark and we com sort of compile benchmark and we com sort of compile benchmark and we com sort of compile this into this into this into a replay agent that just when the tasks a replay agent that just when the tasks a replay agent that just when the tasks arrive replace that sequence of actions arrive replace that sequence of actions arrive replace that sequence of actions blindly. So if you do this for common blindly. So if you do this for common blindly. So if you do this for common benchmark with hundreds of tasks, this benchmark with hundreds of tasks, this benchmark with hundreds of tasks, this is going to be a script that is like is going to be a script that is like is going to be a script that is like less than a megabyte. And this is a less than a megabyte. And this is a less than a megabyte. And this is a completely valid like type of agent that completely valid like type of agent that completely valid like type of agent that you can evaluate on the benchmark. And you can evaluate on the benchmark. And you can evaluate on the benchmark. And if you try to evaluate this uh this kind if you try to evaluate this uh this kind if you try to evaluate this uh this kind of agent on standard benchmarks such as of agent on standard benchmarks such as of agent on standard benchmarks such as OSW word or mobile word, you will see OSW word or mobile word, you will see OSW word or mobile word, you will see that the success rate of this agent that the success rate of this agent that the success rate of this agent compared to the uh frontier model from

compared to the uh frontier model from compared to the uh frontier model from which the agent was extracted is which the agent was extracted is which the agent was extracted is actually uh the same or even better. So actually uh the same or even better. So actually uh the same or even better. So this is a kind of a weird but maybe this is a kind of a weird but maybe this is a kind of a weird but maybe trivial phenomenon but I would argue trivial phenomenon but I would argue trivial phenomenon but I would argue that we shouldn't accept this kind of that we shouldn't accept this kind of that we shouldn't accept this kind of blind scripts beating the frontier blind scripts beating the frontier blind scripts beating the frontier models and the trick here or like the models and the trick here or like the models and the trick here or like the reason for why this happens is that um reason for why this happens is that um reason for why this happens is that um determinism of of the existing of most determinism of of the existing of most determinism of of the existing of most existing benchmarks and so uh if the existing benchmarks and so uh if the existing benchmarks and so uh if the benchmark is static is deterministic benchmark is static is deterministic benchmark is static is deterministic then it is somehow gameable by this sort then it is somehow gameable by this sort then it is somehow gameable by this sort of strategy and it goes even deeper than of strategy and it goes even deeper than of strategy and it goes even deeper than this like if you look at one of the this like if you look at one of the this like if you look at one of the metrics that people have been using in metrics that people have been using in metrics that people have been using in the past for evaluating computer use the past for evaluating computer use the past for evaluating computer use agents um and it's pass at K this metric agents um and it's pass at K this metric agents um and it's pass at K this metric is defined as you know the probability is defined as you know the probability is defined as you know the probability of at least uh one of K attempts of at least uh one of K attempts of at least uh one of K attempts succeeding um but if you look into like succeeding um but if you look into like succeeding um but if you look into like the details of how this metric works on the details of how this metric works on the details of how this metric works on a deterministic environment you will see a deterministic environment you will see a deterministic environment you will see that is literally um and we prove it that is literally um and we prove it that is literally um and we prove it like formally in the paper uh like like formally in the paper uh like like formally in the paper uh like evaluating the success rate of the evaluating the success rate of the evaluating the success rate of the replay agent that I've shown to you. Um replay agent that I've shown to you. Um replay agent that I've shown to you. Um so so basically if that replay agent so so basically if that replay agent so so basically if that replay agent felt weird uh to you also pass at K on felt weird uh to you also pass at K on felt weird uh to you also pass at K on computer use tasks should somehow feel computer use tasks should somehow feel computer use tasks should somehow feel weird to you. Um or in other words pass weird to you. Um or in other words pass weird to you. Um or in other words pass K is sort of a metrification of that uh K is sort of a metrification of that uh K is sort of a metrification of that uh of that exploit of the replay agent. So of that exploit of the replay agent. So of that exploit of the replay agent. So these are two problems specific problems these are two problems specific problems these are two problems specific problems but they point at two general classes of

but they point at two general classes of but they point at two general classes of problems in in Kua benchmarks and uh problems in in Kua benchmarks and uh problems in in Kua benchmarks and uh these problems are around environments. these problems are around environments. these problems are around environments. So building environments and evaluation So building environments and evaluation So building environments and evaluation building good metrics to know if your building good metrics to know if your building good metrics to know if your agent is good or not. uh and in agent is good or not. uh and in agent is good or not. uh and in particular we we want to have particular we we want to have particular we we want to have environments that don't have exploitable environments that don't have exploitable environments that don't have exploitable structure and we want to have metrics structure and we want to have metrics structure and we want to have metrics that are not fragile or they are not that are not fragile or they are not that are not fragile or they are not based on fragile statistics. Uh and so based on fragile statistics. Uh and so based on fragile statistics. Uh and so I'm going to talk about the both of the I'm going to talk about the both of the I'm going to talk about the both of the aspects now. So let's talk about aspects now. So let's talk about aspects now. So let's talk about building principle environments first. building principle environments first. building principle environments first. So the first aspect um that I uh that I So the first aspect um that I uh that I So the first aspect um that I uh that I worked on while uh you know working on worked on while uh you know working on worked on while uh you know working on environments is to try to design a set environments is to try to design a set environments is to try to design a set of principles that could be guiding of principles that could be guiding of principles that could be guiding principle when building environments so principle when building environments so principle when building environments so that build like robust environments and that build like robust environments and that build like robust environments and trustworthy environments. Uh and so if trustworthy environments. Uh and so if trustworthy environments. Uh and so if you think about the problem that I was you think about the problem that I was you think about the problem that I was describing with uh replay agents, uh the describing with uh replay agents, uh the describing with uh replay agents, uh the first thing that you could think about first thing that you could think about first thing that you could think about as a solution not to have a replay agent as a solution not to have a replay agent as a solution not to have a replay agent to like hack your benchmark is uh to to like hack your benchmark is uh to to like hack your benchmark is uh to have your benchmark to be have your benchmark to be have your benchmark to be multiffactorial. So that means uh multiffactorial. So that means uh multiffactorial. So that means uh varying um generating variation for your varying um generating variation for your varying um generating variation for your benchmark. So having stockasticity into benchmark. So having stockasticity into benchmark. So having stockasticity into the benchmark and for computer use the benchmark and for computer use the benchmark and for computer use environment that means uh varying stuff environment that means uh varying stuff environment that means uh varying stuff like uh data or appearance or or simply like uh data or appearance or or simply like uh data or appearance or or simply the initial state. But if you do that the initial state. But if you do that the initial state. But if you do that you want to make sure that all the you want to make sure that all the you want to make sure that all the combinations that you generate are combinations that you generate are combinations that you generate are valid. Uh and so you want to have as a

valid. Uh and so you want to have as a valid. Uh and so you want to have as a design principle in your environment design principle in your environment design principle in your environment also a system for checking and verifying also a system for checking and verifying also a system for checking and verifying that everything is working as as that everything is working as as that everything is working as as intended for every combination. And of intended for every combination. And of intended for every combination. And of course you want uh the usual uh the course you want uh the usual uh the course you want uh the usual uh the usual things for your environment. So usual things for your environment. So usual things for your environment. So you want your environment to be you want your environment to be you want your environment to be sandboxed and you want uh your sandboxed and you want uh your sandboxed and you want uh your environment to support like verifiers. environment to support like verifiers. environment to support like verifiers. So privileged information and you want So privileged information and you want So privileged information and you want of course your environment to be of course your environment to be of course your environment to be realistic. So if it's a reproduction of realistic. So if it's a reproduction of realistic. So if it's a reproduction of a real system, you want uh that a real system, you want uh that a real system, you want uh that reproduction to to be faithful so that reproduction to to be faithful so that reproduction to to be faithful so that the score that you get out is um is a the score that you get out is um is a the score that you get out is um is a good one. Um and so if you sort them out good one. Um and so if you sort them out good one. Um and so if you sort them out uh you can remember these sort of uh you can remember these sort of uh you can remember these sort of principles as the prism principles for principles as the prism principles for principles as the prism principles for for environment design. And we tried for environment design. And we tried for environment design. And we tried that method to build uh a benchmark that that method to build uh a benchmark that that method to build uh a benchmark that would be satisfying all of these would be satisfying all of these would be satisfying all of these principles. And if you look at existing principles. And if you look at existing principles. And if you look at existing benchmarks, some of them do some things benchmarks, some of them do some things benchmarks, some of them do some things uh in a good way. Some of uh some others uh in a good way. Some of uh some others uh in a good way. Some of uh some others do other things in a good way. But there do other things in a good way. But there do other things in a good way. But there is no unified benchmark that sort of is no unified benchmark that sort of is no unified benchmark that sort of matches all of these boxes. And we built matches all of these boxes. And we built matches all of these boxes. And we built one that is called DGword. So the way DG one that is called DGword. So the way DG one that is called DGword. So the way DG word uh in practice was built is as a word uh in practice was built is as a word uh in practice was built is as a set of like mobile apps um for for set of like mobile apps um for for set of like mobile apps um for for Android devices. So it's 15 apps on Android devices. So it's 15 apps on Android devices. So it's 15 apps on spanning different domains uh with 387 spanning different domains uh with 387 spanning different domains uh with 387 uh verified scenarios and uh a number of uh verified scenarios and uh a number of uh verified scenarios and uh a number of configurations. So uh these configurations. So uh these configurations. So uh these configurations they are in a large configurations they are in a large configurations they are in a large number of 3.2 million. Uh but the number of 3.2 million. Uh but the number of 3.2 million. Uh but the important thing is that they are

important thing is that they are important thing is that they are verified. Um and indeed uh the axis are verified. Um and indeed uh the axis are verified. Um and indeed uh the axis are the ones that I was mentioning before. the ones that I was mentioning before. the ones that I was mentioning before. So you can imagine for each one of the So you can imagine for each one of the So you can imagine for each one of the tasks you can vary um things like the tasks you can vary um things like the tasks you can vary um things like the the instance. So what is the exact the instance. So what is the exact the instance. So what is the exact amount of money that you're sending for amount of money that you're sending for amount of money that you're sending for instance or the data profile like which instance or the data profile like which instance or the data profile like which kind of contacts or emails you have in kind of contacts or emails you have in kind of contacts or emails you have in the data for your uh for your task or the data for your uh for your task or the data for your uh for your task or like the theme or the starting screen. like the theme or the starting screen. like the theme or the starting screen. So do you start from the login page or So do you start from the login page or So do you start from the login page or do you start from another valid page? So do you start from another valid page? So do you start from another valid page? So if you do the math, even if you start if you do the math, even if you start if you do the math, even if you start from a relatively low number of uh base from a relatively low number of uh base from a relatively low number of uh base cases for each one of these variables, cases for each one of these variables, cases for each one of these variables, you end up having many many you end up having many many you end up having many many combinations. So you you can get to like combinations. So you you can get to like combinations. So you you can get to like millions of combinations and if you millions of combinations and if you millions of combinations and if you scale this up, you can get to easily to scale this up, you can get to easily to scale this up, you can get to easily to billions of of combinations. Um and all billions of of combinations. Um and all billions of of combinations. Um and all of these uh you know different axis can of these uh you know different axis can of these uh you know different axis can be can be manipulated by coding agents be can be manipulated by coding agents be can be manipulated by coding agents because in the end they are like forms because in the end they are like forms because in the end they are like forms of software. So you can have a coding of software. So you can have a coding of software. So you can have a coding agents to generate different instances agents to generate different instances agents to generate different instances different themes and such. Um, so you different themes and such. Um, so you different themes and such. Um, so you might think maybe it's easy to build an might think maybe it's easy to build an might think maybe it's easy to build an environment. You just generate as much environment. You just generate as much environment. You just generate as much software as you can uh with a coding software as you can uh with a coding software as you can uh with a coding agent and then you have like a diverse agent and then you have like a diverse agent and then you have like a diverse environment. But it's a little bit environment. But it's a little bit environment. But it's a little bit trickier than that. And indeed coding trickier than that. And indeed coding trickier than that. And indeed coding agents can generate a lot of software, agents can generate a lot of software, agents can generate a lot of software, but a lot of software is not the same as but a lot of software is not the same as but a lot of software is not the same as an effective cool environment. And the an effective cool environment. And the an effective cool environment. And the reason for this is that you need to reason for this is that you need to reason for this is that you need to verify the correctness of of your uh

verify the correctness of of your uh verify the correctness of of your uh combination, right? Um and so the the combination, right? Um and so the the combination, right? Um and so the the the key to scale these up is to have a the key to scale these up is to have a the key to scale these up is to have a verification strategy for the variations verification strategy for the variations verification strategy for the variations of of your tasks. Um and so the kind of of of your tasks. Um and so the kind of of of your tasks. Um and so the kind of verification strategy to follow is is verification strategy to follow is is verification strategy to follow is is this one. So you you can generate many this one. So you you can generate many this one. So you you can generate many configs all the combinations uh of the configs all the combinations uh of the configs all the combinations uh of the different factors that I that I've different factors that I that I've different factors that I that I've explained before h and you can then have explained before h and you can then have explained before h and you can then have a system that rejects the broken ones a system that rejects the broken ones a system that rejects the broken ones the ones that are not valid and just the ones that are not valid and just the ones that are not valid and just keeps the the valid configs and so in keeps the the valid configs and so in keeps the the valid configs and so in the case of dig uh we did this by the case of dig uh we did this by the case of dig uh we did this by building a system that looks a little building a system that looks a little building a system that looks a little bit like a compiler and that works in bit like a compiler and that works in bit like a compiler and that works in the in the following way. So you start the in the following way. So you start the in the following way. So you start from a parameterized task template. Um from a parameterized task template. Um from a parameterized task template. Um and so uh this might look like something and so uh this might look like something and so uh this might look like something like this. So you have send a certain like this. So you have send a certain like this. So you have send a certain amount or certain recipient and then you amount or certain recipient and then you amount or certain recipient and then you have a verifier that corresponds to that have a verifier that corresponds to that have a verifier that corresponds to that template and then you have mock data uh template and then you have mock data uh template and then you have mock data uh for that task. So data that you need for for that task. So data that you need for for that task. So data that you need for that specific task to happen. uh and that specific task to happen. uh and that specific task to happen. uh and then we have a system that is like the then we have a system that is like the then we have a system that is like the dig compiler that takes all of these and dig compiler that takes all of these and dig compiler that takes all of these and given a base case of data base case of given a base case of data base case of given a base case of data base case of UI state puts all of these together and UI state puts all of these together and UI state puts all of these together and creates like a valid configuration and creates like a valid configuration and creates like a valid configuration and so you can build systems like this in so you can build systems like this in so you can build systems like this in which the the main craft is good which the the main craft is good which the the main craft is good software engineering uh to make sure software engineering uh to make sure software engineering uh to make sure that actually the combinations that you that actually the combinations that you that actually the combinations that you have are both diverse and valid.

have are both diverse and valid. have are both diverse and valid. And so if you build a benchmark in this And so if you build a benchmark in this And so if you build a benchmark in this way um and you follow the the prism way um and you follow the the prism way um and you follow the the prism principles that I was talking about principles that I was talking about principles that I was talking about before you actually can have a benchmark before you actually can have a benchmark before you actually can have a benchmark that is not hackable in that way. Uh and that is not hackable in that way. Uh and that is not hackable in that way. Uh and so if you do the same process we did so if you do the same process we did so if you do the same process we did before you evaluate your frontier model before you evaluate your frontier model before you evaluate your frontier model and then you evaluate the corresponding and then you evaluate the corresponding and then you evaluate the corresponding replay agent uh you will see that the replay agent uh you will see that the replay agent uh you will see that the replay agent doesn't get a lot of replay agent doesn't get a lot of replay agent doesn't get a lot of performance. Um it gets a little bit of performance. Um it gets a little bit of performance. Um it gets a little bit of performance that is probably what you performance that is probably what you performance that is probably what you want. Sometimes some tasks maybe are want. Sometimes some tasks maybe are want. Sometimes some tasks maybe are repeatable by nature, but on average you repeatable by nature, but on average you repeatable by nature, but on average you you shouldn't expect a replay agent to you shouldn't expect a replay agent to you shouldn't expect a replay agent to have good performance on the benchmark. have good performance on the benchmark. have good performance on the benchmark. Once you build like this diverse um Once you build like this diverse um Once you build like this diverse um combinations, you also can do other combinations, you also can do other combinations, you also can do other things like um measuring the um things like um measuring the um things like um measuring the um robustness of frontier models on robustness of frontier models on robustness of frontier models on different over different axis of different over different axis of different over different axis of variation. So the axis of variation uh I variation. So the axis of variation uh I variation. So the axis of variation uh I described before are here represented described before are here represented described before are here represented there. Um and you can see that in the there. Um and you can see that in the there. Um and you can see that in the worst case frontier models are pretty worst case frontier models are pretty worst case frontier models are pretty bad actually at um being robust to these bad actually at um being robust to these bad actually at um being robust to these variations. And so for instance if you variations. And so for instance if you variations. And so for instance if you have a model uh that seems to be good at have a model uh that seems to be good at have a model uh that seems to be good at a given task, you would expect that if a given task, you would expect that if a given task, you would expect that if you just very you know which screen the you just very you know which screen the you just very you know which screen the task is starting from or like what is task is starting from or like what is task is starting from or like what is the team of the app, the model should the team of the app, the model should the team of the app, the model should pretty much have the same performance.

pretty much have the same performance. pretty much have the same performance. But this is actually not the case for But this is actually not the case for But this is actually not the case for most frontier models. And so if you have most frontier models. And so if you have most frontier models. And so if you have infrastructure like this, you can infrastructure like this, you can infrastructure like this, you can actually measure measure that and like actually measure measure that and like actually measure measure that and like tailor your expectation uh about this tailor your expectation uh about this tailor your expectation uh about this kind of uh robustness. kind of uh robustness. kind of uh robustness. So this was about the first aspect that So this was about the first aspect that So this was about the first aspect that was building an environment that was building an environment that was building an environment that supports um diversity uh and that is supports um diversity uh and that is supports um diversity uh and that is robust enough to evaluate models. But robust enough to evaluate models. But robust enough to evaluate models. But the second aspect is as important as the the second aspect is as important as the the second aspect is as important as the first one is to measure uncertainty first one is to measure uncertainty first one is to measure uncertainty honestly. Uh so once you have all of honestly. Uh so once you have all of honestly. Uh so once you have all of this variation, how do you handle like this variation, how do you handle like this variation, how do you handle like computing um the real performance of computing um the real performance of computing um the real performance of your agent and basically there are two your agent and basically there are two your agent and basically there are two sources of of stoasticity of variation sources of of stoasticity of variation sources of of stoasticity of variation and uh they are not exactly the same but and uh they are not exactly the same but and uh they are not exactly the same but they are equally important and so the they are equally important and so the they are equally important and so the one that we usually think about is the one that we usually think about is the one that we usually think about is the one about the actions right and so you one about the actions right and so you one about the actions right and so you you run your model multiple times you run your model multiple times you run your model multiple times in many cases you can have even quite in many cases you can have even quite in many cases you can have even quite different trajectories out of it because different trajectories out of it because different trajectories out of it because the action at each step would be the action at each step would be the action at each step would be different. uh but if you have a different. uh but if you have a different. uh but if you have a benchmark like the one that I've benchmark like the one that I've benchmark like the one that I've described with multiple combinations described with multiple combinations described with multiple combinations with multiple variations then also the with multiple variations then also the with multiple variations then also the variability from the environment becomes variability from the environment becomes variability from the environment becomes important and we want to capture that important and we want to capture that important and we want to capture that because that is what we are going to because that is what we are going to because that is what we are going to find uh in the real world and so we need find uh in the real world and so we need find uh in the real world and so we need a methodology that captures uh both of a methodology that captures uh both of a methodology that captures uh both of these types of variation these types of variation these types of variation and uh in the paper there are the and uh in the paper there are the and uh in the paper there are the details but but basically we build a

details but but basically we build a details but but basically we build a meth we built a methodology that can meth we built a methodology that can meth we built a methodology that can accurately um capture these two types of accurately um capture these two types of accurately um capture these two types of variation taking into account the variation taking into account the variation taking into account the structure of the benchmark and so if you structure of the benchmark and so if you structure of the benchmark and so if you start like in in practice it is useful start like in in practice it is useful start like in in practice it is useful to use this concept of coverage. Uh so to use this concept of coverage. Uh so to use this concept of coverage. Uh so when you when you compute a confidence when you when you compute a confidence when you when you compute a confidence interval uh basically you have some interval uh basically you have some interval uh basically you have some confidence that the performance of the confidence that the performance of the confidence that the performance of the model is inside of that range. Um and so model is inside of that range. Um and so model is inside of that range. Um and so you would expect that a 95% confidence you would expect that a 95% confidence you would expect that a 95% confidence interval would say that you know 95% of interval would say that you know 95% of interval would say that you know 95% of the time the performance of the model is the time the performance of the model is the time the performance of the model is on that range. But if you only use on that range. But if you only use on that range. But if you only use rollouts, so you only use the base case rollouts, so you only use the base case rollouts, so you only use the base case and what people would use normally and what people would use normally and what people would use normally actually in realistic cases you have actually in realistic cases you have actually in realistic cases you have something like 17% on 20% coverage. And something like 17% on 20% coverage. And something like 17% on 20% coverage. And that means that basically you only 20% that means that basically you only 20% that means that basically you only 20% of the time you uh guess the right of the time you uh guess the right of the time you uh guess the right performance of the agent which can be performance of the agent which can be performance of the agent which can be pretty bad. But like if you take it into pretty bad. But like if you take it into pretty bad. But like if you take it into into account the hierarchy and you use into account the hierarchy and you use into account the hierarchy and you use the proper way of computing confidence the proper way of computing confidence the proper way of computing confidence intervals, you can get to the to the intervals, you can get to the to the intervals, you can get to the to the full confidence interval and be 90 95% full confidence interval and be 90 95% full confidence interval and be 90 95% accurate. And so if this seems quite accurate. And so if this seems quite accurate. And so if this seems quite abstract uh you know in practice that abstract uh you know in practice that abstract uh you know in practice that means that if you want to make a means that if you want to make a means that if you want to make a decision about which models to deploy decision about which models to deploy decision about which models to deploy maybe you have model A model B and you maybe you have model A model B and you maybe you have model A model B and you do an eval for for those two models um do an eval for for those two models um do an eval for for those two models um you can have cases in which the you can have cases in which the you can have cases in which the confidence intervals seem really really confidence intervals seem really really confidence intervals seem really really small uh and so you make a decision

small uh and so you make a decision small uh and so you make a decision based on those small confidence based on those small confidence based on those small confidence intervals but they are actually intervals but they are actually intervals but they are actually overconfident and so uh this was the overconfident and so uh this was the overconfident and so uh this was the wrong decision. So the the orange bars wrong decision. So the the orange bars wrong decision. So the the orange bars are the real performance here. Uh so you are the real performance here. Uh so you are the real performance here. Uh so you make this decision and if a mistake is make this decision and if a mistake is make this decision and if a mistake is pretty costly for you and you have many pretty costly for you and you have many pretty costly for you and you have many tasks like if you have one million tasks tasks like if you have one million tasks tasks like if you have one million tasks and there is a 4% mismatch in and there is a 4% mismatch in and there is a 4% mismatch in performance for real in the models and performance for real in the models and performance for real in the models and each mistake is like $20 uh like $12 on each mistake is like $20 uh like $12 on each mistake is like $20 uh like $12 on average. Uh it can cost you like average. Uh it can cost you like average. Uh it can cost you like hundreds of thousands of dollars uh in a hundreds of thousands of dollars uh in a hundreds of thousands of dollars uh in a single month. So it can be like super single month. So it can be like super single month. So it can be like super costly as a mistake just for a costly as a mistake just for a costly as a mistake just for a confidence interval being overconfident. confidence interval being overconfident. confidence interval being overconfident. But if you have like a reliable way of But if you have like a reliable way of But if you have like a reliable way of computing the confidence interval, the computing the confidence interval, the computing the confidence interval, the method would tell to you um I'm not method would tell to you um I'm not method would tell to you um I'm not confident enough to make an informed confident enough to make an informed confident enough to make an informed decision. And so you can choose like to decision. And so you can choose like to decision. And so you can choose like to spend more money to spend more time on spend more money to spend more time on spend more money to spend more time on evaluating models and avoid the co evaluating models and avoid the co evaluating models and avoid the co costly mistake. So we don't want to like costly mistake. So we don't want to like costly mistake. So we don't want to like elude ourselves with like wrong elude ourselves with like wrong elude ourselves with like wrong confidence intervals because there's uh confidence intervals because there's uh confidence intervals because there's uh you know money on the table essentially.

you know money on the table essentially. you know money on the table essentially. And so this is sort of a final checklist And so this is sort of a final checklist And so this is sort of a final checklist of uh the things that I've uh that I've of uh the things that I've uh that I've of uh the things that I've uh that I've discussed so far. Uh so again to recap discussed so far. Uh so again to recap discussed so far. Uh so again to recap some of the things that are important in some of the things that are important in some of the things that are important in building on benchmark are about the building on benchmark are about the building on benchmark are about the environment and some other things are environment and some other things are environment and some other things are about the metrics. So on the environment about the metrics. So on the environment about the metrics. So on the environment you can follow uh the principles uh that you can follow uh the principles uh that you can follow uh the principles uh that I described before like prism I described before like prism I described before like prism principles. So some of these things are principles. So some of these things are principles. So some of these things are rather common but some things like um rather common but some things like um rather common but some things like um varying initial state across runs they varying initial state across runs they varying initial state across runs they are pretty rare across existing are pretty rare across existing are pretty rare across existing benchmarks but they are very important benchmarks but they are very important benchmarks but they are very important and so I would suggest you to try to and so I would suggest you to try to and so I would suggest you to try to incorporate these into your evals and incorporate these into your evals and incorporate these into your evals and and things about the metrics. Uh you can and things about the metrics. Uh you can and things about the metrics. Uh you can of course read the paper for the details of course read the paper for the details of course read the paper for the details but essentially um it's very important but essentially um it's very important but essentially um it's very important to avoid uh replayability as something to avoid uh replayability as something to avoid uh replayability as something that uh you can have in your benchmark that uh you can have in your benchmark that uh you can have in your benchmark and also to focus on having accurate and also to focus on having accurate and also to focus on having accurate confidence intervals. Uh so respecting confidence intervals. Uh so respecting confidence intervals. Uh so respecting the benchmark structure and trying to the benchmark structure and trying to the benchmark structure and trying to avoid um underestimating the uncertainty avoid um underestimating the uncertainty avoid um underestimating the uncertainty overall.

overall. overall. And so I've heard many times sentences And so I've heard many times sentences And so I've heard many times sentences like this. this benchmark can be gamed like this. this benchmark can be gamed like this. this benchmark can be gamed but everybody's still using it or like but everybody's still using it or like but everybody's still using it or like there is no error bar but I don't see there is no error bar but I don't see there is no error bar but I don't see people using them. So these are sort of people using them. So these are sort of people using them. So these are sort of uh things that we can think when we uh things that we can think when we uh things that we can think when we don't have enough time but actually a don't have enough time but actually a don't have enough time but actually a non-rigorous benchmark is misleading. Um non-rigorous benchmark is misleading. Um non-rigorous benchmark is misleading. Um you know it can be misleading for the you know it can be misleading for the you know it can be misleading for the field because everybody could be seeking field because everybody could be seeking field because everybody could be seeking you know um maximizing a score on a you know um maximizing a score on a you know um maximizing a score on a benchmark that maybe is not capturing benchmark that maybe is not capturing benchmark that maybe is not capturing what we care about but especially it can what we care about but especially it can what we care about but especially it can be misleading for uh you know your own be misleading for uh you know your own be misleading for uh you know your own decisions. And so if you um are deluding decisions. And so if you um are deluding decisions. And so if you um are deluding yourself on thinking that a score is yourself on thinking that a score is yourself on thinking that a score is like confident and that is confidently like confident and that is confidently like confident and that is confidently telling that your model is good actually telling that your model is good actually telling that your model is good actually you are going to pay for those mistakes you are going to pay for those mistakes you are going to pay for those mistakes and so I think it's very good uh usually and so I think it's very good uh usually and so I think it's very good uh usually to be honest with with yourself and to to be honest with with yourself and to to be honest with with yourself and to try to be rigorous in the evaluations try to be rigorous in the evaluations try to be rigorous in the evaluations that you have. Um as the last slide uh I that you have. Um as the last slide uh I that you have. Um as the last slide uh I just started uh this company programmer just started uh this company programmer just started uh this company programmer and we are building uh the best and we are building uh the best and we are building uh the best infrastructure for ku enable infrastructure for ku enable infrastructure for ku enable verification uh and so we are hiring if verification uh and so we are hiring if verification uh and so we are hiring if you're interested or want to chat and you're interested or want to chat and you're interested or want to chat and this is our website. Thank you very this is our website. Thank you very this is our website. Thank you very much. [applause]