Benchmarks: The Good, the Bad, and the Ugly — Ali Khial, G2i

Hello everyone. Hello everyone. Um Um Um this is the last talk of this session. this is the last talk of this session. this is the last talk of this session. So hopefully it's going to be short. I So hopefully it's going to be short. I So hopefully it's going to be short. I know that you guys had to go through a know that you guys had to go through a know that you guys had to go through a long day. So long day. So long day. So try to keep it short and light for you try to keep it short and light for you try to keep it short and light for you all. all. all. Um I'm going to present myself. Um I'm Um I'm going to present myself. Um I'm Um I'm going to present myself. Um I'm Ali. I'm the director of AI and ML at Ali. I'm the director of AI and ML at Ali. I'm the director of AI and ML at G2I. G2I. G2I. Um I have zero experience in ML. So I Um I have zero experience in ML. So I Um I have zero experience in ML. So I don't know why they put the ML in my don't know why they put the ML in my don't know why they put the ML in my title. I'm a software engineer uh at title. I'm a software engineer uh at title. I'm a software engineer uh at heart. And to prove that I have more heart. And to prove that I have more heart. And to prove that I have more than 50 abandoned side projects in my than 50 abandoned side projects in my than 50 abandoned side projects in my machine. So machine. So machine. So uh you can know. uh you can know. uh you can know. So uh I'm going to make a disclaimer. So uh I'm going to make a disclaimer. So uh I'm going to make a disclaimer. The the title of the the presentation is The the title of the the presentation is The the title of the the presentation is a little bit misleading. Uh as I was a little bit misleading. Uh as I was a little bit misleading. Uh as I was working on it, I realized that it would working on it, I realized that it would working on it, I realized that it would be better if I presented my journey be better if I presented my journey be better if I presented my journey uh into benchmarks and what I learned uh into benchmarks and what I learned uh into benchmarks and what I learned instead of trying to find a dichotomy of instead of trying to find a dichotomy of instead of trying to find a dichotomy of the the the the bad, the ugly, and and the good. So the bad, the ugly, and and the good. So the bad, the ugly, and and the good. So um let's start with um um let's start with um um let's start with um I want to grab your attention. I want to grab your attention. I want to grab your attention. And I invite you to look at this. And I invite you to look at this. And I invite you to look at this. These beautiful three screenshots are a These beautiful three screenshots are a These beautiful three screenshots are a single prompt on one of the benchmark single prompt on one of the benchmark single prompt on one of the benchmark tasks.

tasks. tasks. As I was looking at it, I was like As I was looking at it, I was like As I was looking at it, I was like how can an engineer write a task like how can an engineer write a task like how can an engineer write a task like this? So I said, "Nah, it's impossible. this? So I said, "Nah, it's impossible. this? So I said, "Nah, it's impossible. No one writes prompts like these ever." No one writes prompts like these ever." No one writes prompts like these ever." But I wanted to double-check with my But I wanted to double-check with my But I wanted to double-check with my engineers. So I took three of our best engineers. So I took three of our best engineers. So I took three of our best engineers. I showed them the prompt and engineers. I showed them the prompt and engineers. I showed them the prompt and I said, "Would you ever write a prompt I said, "Would you ever write a prompt I said, "Would you ever write a prompt like this?" And the answer was like this?" And the answer was like this?" And the answer was no. And they're right. They shouldn't. And they're right. They shouldn't. And so And so And so at that point I'm I was like at that point I'm I was like at that point I'm I was like what is a what are benchmarks anyway? Uh what is a what are benchmarks anyway? Uh what is a what are benchmarks anyway? Uh I needed to take a step back. I needed I needed to take a step back. I needed I needed to take a step back. I needed to look more. I needed to understand. to look more. I needed to understand. to look more. I needed to understand. And so And so And so as I was researching, I faced a wall of as I was researching, I faced a wall of as I was researching, I faced a wall of keywords. keywords. keywords. Um Um Um graders, long horizon, verifiers, bench graders, long horizon, verifiers, bench graders, long horizon, verifiers, bench benchmarks, and and a lot of jargon. So, benchmarks, and and a lot of jargon. So, benchmarks, and and a lot of jargon. So, I was like, either this is too I was like, either this is too I was like, either this is too complicated or complicated or complicated or um um um there's a lot of jargon and a lot of um there's a lot of jargon and a lot of um there's a lot of jargon and a lot of um words to to to work through here. So, um words to to to work through here. So, um words to to to work through here. So, um I worked through it, worked with my I worked through it, worked with my I worked through it, worked with my team. I have a lot of good researchers team. I have a lot of good researchers team. I have a lot of good researchers in the team, and we uh kind of like in the team, and we uh kind of like in the team, and we uh kind of like nailed like simplified it to the most nailed like simplified it to the most nailed like simplified it to the most basics. Um and so, basics. Um and so, basics. Um and so, the way I see it is that it starts as a the way I see it is that it starts as a the way I see it is that it starts as a prompt or an instruction.

prompt or an instruction. prompt or an instruction. That prompt is fed to models and agents. Agents provide solutions. Agents provide solutions. Those solutions are verified uh and Those solutions are verified uh and Those solutions are verified uh and graded through verifiers and rubrics. graded through verifiers and rubrics. graded through verifiers and rubrics. All of that is wrapped in a harness All of that is wrapped in a harness All of that is wrapped in a harness that's that's preventing it from um that's that's preventing it from um that's that's preventing it from um from the external factors. And if it all from the external factors. And if it all from the external factors. And if it all goes good, uh we have um goes good, uh we have um goes good, uh we have um trajectories, scores, trajectories, scores, trajectories, scores, and um and um and um metadata that we can use metadata that we can use metadata that we can use um to to to verif- to um to to to verif- to um to to to verif- to basically uh rank um models. basically uh rank um models. basically uh rank um models. And so, the equation is simple. And so, the equation is simple. And so, the equation is simple. If prompts and instructions are great If prompts and instructions are great If prompts and instructions are great and verifiers and rubrics are doing and verifiers and rubrics are doing and verifiers and rubrics are doing their job while the harness is their job while the harness is their job while the harness is preventing preventing preventing um or creating an environment that is um or creating an environment that is um or creating an environment that is good for a benchmark, good for a benchmark, good for a benchmark, we should have amazing results. we should have amazing results. we should have amazing results. Um but, that's not the reality. So, what Um but, that's not the reality. So, what Um but, that's not the reality. So, what what what went wrong?

So, the first thing is So, the first thing is when looking deeper in benchmarks, uh when looking deeper in benchmarks, uh when looking deeper in benchmarks, uh most of most of most of the instructions are unrealistic. the instructions are unrealistic. the instructions are unrealistic. Um I did a quick research on SweetBench Um I did a quick research on SweetBench Um I did a quick research on SweetBench Pro, and um Pro, and um Pro, and um there's 481 there's 481 there's 481 words words words per instruction in average. That's a per instruction in average. That's a per instruction in average. That's a two-pager per task. two-pager per task. two-pager per task. That is not how people write prompts. That is not how people write prompts. That is not how people write prompts. And to illustrate more of that, And to illustrate more of that, And to illustrate more of that, um I took a couple examples here. um I took a couple examples here. um I took a couple examples here. The first one I looked at I I call the The first one I looked at I I call the The first one I looked at I I call the leaky prompt. It's a go um task that's leaky prompt. It's a go um task that's leaky prompt. It's a go um task that's basically um that's trying to match in basically um that's trying to match in basically um that's trying to match in some rejects and doing test it's on on some rejects and doing test it's on on some rejects and doing test it's on on some rejects. So, in the first some rejects. So, in the first some rejects. So, in the first screenshot here, screenshot here, screenshot here, um the instruction is pointing directly um the instruction is pointing directly um the instruction is pointing directly to the test file, to the test file, to the test file, which basically means that the LLM has which basically means that the LLM has which basically means that the LLM has all the ingredient it needs to go and all the ingredient it needs to go and all the ingredient it needs to go and find that test file and implement based find that test file and implement based find that test file and implement based on that. The second one is on that. The second one is on that. The second one is is even worse. Um is even worse. Um is even worse. Um it's basically providing a complete it's basically providing a complete it's basically providing a complete interface of the implementation.

interface of the implementation. interface of the implementation. Basically locking the LLM from any kind Basically locking the LLM from any kind Basically locking the LLM from any kind of of of uh creativity and it's forcing it to do uh creativity and it's forcing it to do uh creativity and it's forcing it to do it that way. So, it that way. So, it that way. So, that's the leaky prompt. that's the leaky prompt. that's the leaky prompt. The second example, it's the The second example, it's the The second example, it's the the not economically valuable prompt. Uh the not economically valuable prompt. Uh the not economically valuable prompt. Uh this is from Sweet Marathon. this is from Sweet Marathon. this is from Sweet Marathon. And this prompt is And this prompt is And this prompt is well-formed. well-formed. well-formed. It's It's It's It's It's It's It's abstracted enough to allow for the It's abstracted enough to allow for the It's abstracted enough to allow for the LLM to do its work, but it's asking it LLM to do its work, but it's asking it LLM to do its work, but it's asking it to build a C compiler in Rust. So, I to build a C compiler in Rust. So, I to build a C compiler in Rust. So, I don't know if any of you ever tried to don't know if any of you ever tried to don't know if any of you ever tried to do that, but I don't think it's a good do that, but I don't think it's a good do that, but I don't think it's a good idea. We should not do that. All right, moving on. All right, moving on. The second problem, weak verifiers. The second problem, weak verifiers. The second problem, weak verifiers. Um so, the screenshot here is is a uh is Um so, the screenshot here is is a uh is Um so, the screenshot here is is a uh is the work that Deep Sweet the work that Deep Sweet the work that Deep Sweet um did um did um did uh to compare their uh their bench uh to compare their uh their bench uh to compare their uh their bench against Sweet Bench Pro. against Sweet Bench Pro. against Sweet Bench Pro. And um And um And um let me just fix here so I can see the let me just fix here so I can see the let me just fix here so I can see the numbers.

In Sweet Bench Pro, 8.5 of 8.5% of all In Sweet Bench Pro, 8.5 of 8.5% of all the tasks uh the tasks uh the tasks uh accepted wrong implementation in one accepted wrong implementation in one accepted wrong implementation in one hand and more than 20 24% of the tasks hand and more than 20 24% of the tasks hand and more than 20 24% of the tasks uh rejected um uh rejected um uh rejected um correct implementations. correct implementations. correct implementations. And so, I kind of went again, dug a And so, I kind of went again, dug a And so, I kind of went again, dug a little bit, and I extracted one of the little bit, and I extracted one of the little bit, and I extracted one of the tasks, and I started looking at it. tasks, and I started looking at it. tasks, and I started looking at it. Um and and here's here's what's Um and and here's here's what's Um and and here's here's what's happening in the example of uh happening in the example of uh happening in the example of uh re- rejecting um re- rejecting um re- rejecting um possibly rejecting good good answers. possibly rejecting good good answers. possibly rejecting good good answers. So, in this example, the test is is So, in this example, the test is is So, in this example, the test is is basically expecting a variable to exist. basically expecting a variable to exist. basically expecting a variable to exist. But that variable is first not specified But that variable is first not specified But that variable is first not specified in the instruction, and two, in the instruction, and two, in the instruction, and two, why would we expect an LLM to write the why would we expect an LLM to write the why would we expect an LLM to write the variable name this way? So, this test is variable name this way? So, this test is variable name this way? So, this test is cornering the LLM cornering the LLM cornering the LLM and basically uh and basically uh and basically uh causing uh those false negatives. In the causing uh those false negatives. In the causing uh those false negatives. In the other example, other example, other example, it's base the test is basically it's base the test is basically it's base the test is basically checking functions that are unexported.

checking functions that are unexported. checking functions that are unexported. So, if that was a PR in any of our So, if that was a PR in any of our So, if that was a PR in any of our projects, projects, projects, and exposed these type of tests, we and exposed these type of tests, we and exposed these type of tests, we would not accept it. So, this is would not accept it. So, this is would not accept it. So, this is what a weak verifier looks like. All right, moving on. All right, moving on. Re- reward hacking. So, what's happening Re- reward hacking. So, what's happening Re- reward hacking. So, what's happening is models are becoming increasingly is models are becoming increasingly is models are becoming increasingly increasingly able to optimize increasingly able to optimize increasingly able to optimize and figure out solutions to hard and figure out solutions to hard and figure out solutions to hard problems by going around the problem. problems by going around the problem. problems by going around the problem. So, instead of actually trying to fix So, instead of actually trying to fix So, instead of actually trying to fix the to to apply a patch to a task, they the to to apply a patch to a task, they the to to apply a patch to a task, they try to go and find dot git folders, or try to go and find dot git folders, or try to go and find dot git folders, or they look up the internet for any kind they look up the internet for any kind they look up the internet for any kind of traces that would allow them to um to of traces that would allow them to um to of traces that would allow them to um to do the task. do the task. do the task. And this first graph here shows like And this first graph here shows like And this first graph here shows like shows that as models evolve, they are shows that as models evolve, they are shows that as models evolve, they are now more smarter and smarter in being now more smarter and smarter in being now more smarter and smarter in being able to do reward hacking, but able to do reward hacking, but able to do reward hacking, but that's what we want. We want LLMs to be that's what we want. We want LLMs to be that's what we want. We want LLMs to be smart. The benchmarks are lacking behind smart. The benchmarks are lacking behind smart. The benchmarks are lacking behind and they're not preventing from from and they're not preventing from from and they're not preventing from from that to happen.

that to happen. that to happen. Um Um Um More in detail, as you can see here, More in detail, as you can see here, More in detail, as you can see here, the more you go in time and the more you the more you go in time and the more you the more you go in time and the more you have new versions, the delta of um have new versions, the delta of um have new versions, the delta of um of um of um of um reward hacking is increasing. So, the conclusion here is So, the conclusion here is there's a quality gap and it's causing a there's a quality gap and it's causing a there's a quality gap and it's causing a trust gap. trust gap. trust gap. I have not met an engineer in the last 6 I have not met an engineer in the last 6 I have not met an engineer in the last 6 months that would choose a model or months that would choose a model or months that would choose a model or choose um an LLM based on the choose um an LLM based on the choose um an LLM based on the leaderboards. leaderboards. leaderboards. Um they look at them. Um they look at them. Um they look at them. There's a lot of hype, but then they There's a lot of hype, but then they There's a lot of hype, but then they move on and they test things by move on and they test things by move on and they test things by themselves themselves themselves and they apply that. So, how do we close the gap? So, how do we close the gap? Um Um Um in the last 2 months, we've been working in the last 2 months, we've been working in the last 2 months, we've been working with our team at G2i to basically with our team at G2i to basically with our team at G2i to basically try to define a framework, uh a set of try to define a framework, uh a set of try to define a framework, uh a set of principles that would allow us to principles that would allow us to principles that would allow us to build tasks for benchmarks build tasks for benchmarks build tasks for benchmarks that are um better than what we have that are um better than what we have that are um better than what we have today.

The first one, The first one, human instructions. human instructions. human instructions. Authored by humans, reviewed by humans. Authored by humans, reviewed by humans. Authored by humans, reviewed by humans. This is basically the entry point for This is basically the entry point for This is basically the entry point for any great tasks. The instructions given any great tasks. The instructions given any great tasks. The instructions given to an agent or an LLM should lean to an agent or an LLM should lean to an agent or an LLM should lean towards expressing desired behaviors, towards expressing desired behaviors, towards expressing desired behaviors, objectives, and hard constraints, not objectives, and hard constraints, not objectives, and hard constraints, not implement details implement details implement details or try to guarantee self-containment or try to guarantee self-containment or try to guarantee self-containment when when when the task itself is is expressing too the task itself is is expressing too the task itself is is expressing too much uh much uh much uh too much details. The second principle is holistic The second principle is holistic graders. graders. graders. Behavioral tests in one hand and then Behavioral tests in one hand and then Behavioral tests in one hand and then precision what were needed. This is very precision what were needed. This is very precision what were needed. This is very similar to how we approach similar to how we approach similar to how we approach um tests in engineering. um tests in engineering. um tests in engineering. We want to have the most surface covered We want to have the most surface covered We want to have the most surface covered without being too prescriptive, but we without being too prescriptive, but we without being too prescriptive, but we also want to be precise where needed. also want to be precise where needed. also want to be precise where needed. So, for security issues or business So, for security issues or business So, for security issues or business logic, we want to have logic, we want to have logic, we want to have the whole stack units unit test the whole stack units unit test the whole stack units unit test integration tests and then end-to-end integration tests and then end-to-end integration tests and then end-to-end tests. But, for the rest of the the rest tests. But, for the rest of the the rest tests. But, for the rest of the the rest of the the software, of the the software, of the the software, we don't want to have 100% coverage we don't want to have 100% coverage we don't want to have 100% coverage because that's um not efficient.

The third principle, production grade. The third principle, production grade. The tasks have to be The tasks have to be The tasks have to be tasks have to have value tasks have to have value tasks have to have value um um um and they have to be economically and they have to be economically and they have to be economically valuable. Um valuable. Um valuable. Um it is one thing to have a test a task it is one thing to have a test a task it is one thing to have a test a task that is failing the LLM proven that the that is failing the LLM proven that the that is failing the LLM proven that the LLM is not there yet. LLM is not there yet. LLM is not there yet. It is another for it's another thing for It is another for it's another thing for It is another for it's another thing for an engineer to look at a task and say, an engineer to look at a task and say, an engineer to look at a task and say, "If the LLM is fixing this, I trust it "If the LLM is fixing this, I trust it "If the LLM is fixing this, I trust it to fix that." to fix that." to fix that." Currently, we don't have that. So, Currently, we don't have that. So, Currently, we don't have that. So, production grade. The fourth principle, contamination free The fourth principle, contamination free by design. by design. by design. We want to do novel tasks only We want to do novel tasks only We want to do novel tasks only and we want to make sure that we keep and we want to make sure that we keep and we want to make sure that we keep private holdout sets. private holdout sets. private holdout sets. This is This is This is a principle that is very important as a principle that is very important as a principle that is very important as currently currently currently the tasks that are existing in the tasks that are existing in the tasks that are existing in benchmarks benchmarks benchmarks are all put from GitHub repos or from um are all put from GitHub repos or from um are all put from GitHub repos or from um from from public repos. So, our approach from from public repos. So, our approach from from public repos. So, our approach here is that it should always be novel.

here is that it should always be novel. here is that it should always be novel. This way, it's contamination free by This way, it's contamination free by This way, it's contamination free by design. design. design. And the fifth and last principle here And the fifth and last principle here And the fifth and last principle here is information about leaderboards. is information about leaderboards. is information about leaderboards. The benchmark needs to tell a story and The benchmark needs to tell a story and The benchmark needs to tell a story and needs to help people make decisions. needs to help people make decisions. needs to help people make decisions. Leaderboards are what we see in Leaderboards are what we see in Leaderboards are what we see in benchmarks today. benchmarks today. benchmarks today. They tell you who wins, They tell you who wins, They tell you who wins, but they don't to you why. And so, we but they don't to you why. And so, we but they don't to you why. And so, we want to basically put the x-axis back on want to basically put the x-axis back on want to basically put the x-axis back on um on on the first page. um on on the first page. um on on the first page. Uh the idea here is that there's um there's a lot of um there's a lot of um data that we can extract from those data that we can extract from those data that we can extract from those these runs, and unfortunately, they're these runs, and unfortunately, they're these runs, and unfortunately, they're not being put in the forefront. not being put in the forefront. not being put in the forefront. And people have to dig uh a lot and do And people have to dig uh a lot and do And people have to dig uh a lot and do their own experiments to get to those their own experiments to get to those their own experiments to get to those data points. And so, finally, uh initially, I wanted And so, finally, uh initially, I wanted to to to have a kind of a a lofty like ending to have a kind of a a lofty like ending to have a kind of a a lofty like ending to this, but I think I I I pivoted to this, but I think I I I pivoted to this, but I think I I I pivoted to something more interesting. Uh this is a something more interesting. Uh this is a something more interesting. Uh this is a call to action to software engineers.

call to action to software engineers. call to action to software engineers. Um Um Um benchmarks are not hard. We need to look benchmarks are not hard. We need to look benchmarks are not hard. We need to look under the hood. under the hood. under the hood. And we need to understand them and join And we need to understand them and join And we need to understand them and join the Discord because engineers' input is the Discord because engineers' input is the Discord because engineers' input is valuable.