DeepSWE: A Contamination-Resistant Coding Benchmark — James Shi, Datacurve

Hey everyone, can you guys hear me? Hey everyone, can you guys hear me? Okay, this is good. Um, yeah, my name is Okay, this is good. Um, yeah, my name is Okay, this is good. Um, yeah, my name is James. I'm one of the founding engineers James. I'm one of the founding engineers James. I'm one of the founding engineers at Data Curve. Unfortunately, Serena's at Data Curve. Unfortunately, Serena's at Data Curve. Unfortunately, Serena's been out with a fever for the past been out with a fever for the past been out with a fever for the past couple of days. Um, she was supposed to couple of days. Um, she was supposed to couple of days. Um, she was supposed to be here giving this talk. Um, so I'm be here giving this talk. Um, so I'm be here giving this talk. Um, so I'm just filling in uh in her place, but just filling in uh in her place, but just filling in uh in her place, but I've been at uh data curve working on I've been at uh data curve working on I've been at uh data curve working on the research and engineering side of the research and engineering side of the research and engineering side of things. Um, as well as Deep Suite, which things. Um, as well as Deep Suite, which things. Um, as well as Deep Suite, which is our uh frontier long horizon coding is our uh frontier long horizon coding is our uh frontier long horizon coding benchmark, which you guys may be benchmark, which you guys may be benchmark, which you guys may be familiar. Um, I'll just be going over, familiar. Um, I'll just be going over, familiar. Um, I'll just be going over, you know, some of the most important you know, some of the most important you know, some of the most important findings about Deep Suite. Um, a brief findings about Deep Suite. Um, a brief findings about Deep Suite. Um, a brief overview of what it is for those of you overview of what it is for those of you overview of what it is for those of you who don't who who may not know. um and who don't who who may not know. um and who don't who who may not know. um and then going deeper into our methodology then going deeper into our methodology then going deeper into our methodology and exactly how we came about this uh uh and exactly how we came about this uh uh and exactly how we came about this uh uh frontier coding benchmark. So um deepu frontier coding benchmark. So um deepu frontier coding benchmark. So um deepu is a long horizon software engineering is a long horizon software engineering is a long horizon software engineering software engineering benchmark comprised software engineering benchmark comprised software engineering benchmark comprised of 113 original software engineering of 113 original software engineering of 113 original software engineering tasks. So this means unlike something tasks. So this means unlike something tasks. So this means unlike something like sweet bench pro we didn't scrape like sweet bench pro we didn't scrape like sweet bench pro we didn't scrape this from um existing PRs that have been this from um existing PRs that have been this from um existing PRs that have been closed. Um there's a variety of benefits closed. Um there's a variety of benefits closed. Um there's a variety of benefits for this. Uh namely one of them is to for this. Uh namely one of them is to for this. Uh namely one of them is to resist against contamination and agents resist against contamination and agents resist against contamination and agents being able to cheat uh through the being able to cheat uh through the being able to cheat uh through the course of their rollouts. Um Swebench course of their rollouts. Um Swebench course of their rollouts. Um Swebench has or Swebench Pro uh pulls thousands has or Swebench Pro uh pulls thousands has or Swebench Pro uh pulls thousands of tasks from only 40 repositories. The of tasks from only 40 repositories. The of tasks from only 40 repositories. The median task per repository for us is median task per repository for us is median task per repository for us is one. So you can see across over a 100 one. So you can see across over a 100 one. So you can see across over a 100 tasks we pull from nearly 100 tasks we pull from nearly 100 tasks we pull from nearly 100 repositories. And the language spans repositories. And the language spans repositories. And the language spans across uh Typescript, JavaScript, across uh Typescript, JavaScript, across uh Typescript, JavaScript, Python, Rust, and Go. And we have plans Python, Rust, and Go. And we have plans Python, Rust, and Go. And we have plans to add uh more languages later on. Um

to add uh more languages later on. Um to add uh more languages later on. Um since its uh release, we've received uh since its uh release, we've received uh since its uh release, we've received uh very positive reception. It's replaced very positive reception. It's replaced very positive reception. It's replaced SweetBench Pro in the artificial SweetBench Pro in the artificial SweetBench Pro in the artificial analysis uh coding um agent index as analysis uh coding um agent index as analysis uh coding um agent index as well as being cited by uh numerous well as being cited by uh numerous well as being cited by uh numerous Frontier model labs and us helping with Frontier model labs and us helping with Frontier model labs and us helping with them in uh tracking their models on our them in uh tracking their models on our them in uh tracking their models on our benchmark as well. So, we've been really benchmark as well. So, we've been really benchmark as well. So, we've been really really appreciative of that. really appreciative of that. really appreciative of that. Uh bit of context about us. Data curve Uh bit of context about us. Data curve Uh bit of context about us. Data curve works on building uh training data for works on building uh training data for works on building uh training data for high ceiling domains including coding as high ceiling domains including coding as high ceiling domains including coding as well as coding adjacent fields. Uh we well as coding adjacent fields. Uh we well as coding adjacent fields. Uh we also are trying to answer the very uh also are trying to answer the very uh also are trying to answer the very uh elusive question of what exactly makes elusive question of what exactly makes elusive question of what exactly makes uh good data, what is data quality and uh good data, what is data quality and uh good data, what is data quality and how can we demonstrate that our training how can we demonstrate that our training how can we demonstrate that our training data in fact moves the needle. So Deep data in fact moves the needle. So Deep data in fact moves the needle. So Deep Suite is one in a long line of Suite is one in a long line of Suite is one in a long line of initiatives that we have towards initiatives that we have towards initiatives that we have towards answering this question. answering this question. answering this question. So why did we create Deep Suite? Well, So why did we create Deep Suite? Well, So why did we create Deep Suite? Well, it was very clear that the existing it was very clear that the existing it was very clear that the existing benchmarks are not hitting the mark. Um benchmarks are not hitting the mark. Um benchmarks are not hitting the mark. Um with benches like Sweetbench Pro, uh top with benches like Sweetbench Pro, uh top with benches like Sweetbench Pro, uh top models are clustering at the top. It's models are clustering at the top. It's models are clustering at the top. It's very hard to differentiate between uh very hard to differentiate between uh very hard to differentiate between uh which one is good because they all have which one is good because they all have which one is good because they all have overlapping confidence intervals. uh overlapping confidence intervals. uh overlapping confidence intervals. uh contamination is also rampant because contamination is also rampant because contamination is also rampant because again all of these tasks are mined from again all of these tasks are mined from again all of these tasks are mined from public PRs. So all the solution tests public PRs. So all the solution tests public PRs. So all the solution tests even the discussion around the PRs those even the discussion around the PRs those even the discussion around the PRs those are all available out in the wild for are all available out in the wild for are all available out in the wild for these agents to access. The verifiers these agents to access. The verifiers these agents to access. The verifiers are also very very brittle because we're are also very very brittle because we're are also very very brittle because we're anchoring them to a specific anchoring them to a specific anchoring them to a specific implementation often derived from the PR

implementation often derived from the PR implementation often derived from the PR that was merged in. And oftentimes you that was merged in. And oftentimes you that was merged in. And oftentimes you also have tests that check for private also have tests that check for private also have tests that check for private helpers and functions uh created by the helpers and functions uh created by the helpers and functions uh created by the task author which is uh very opinionated task author which is uh very opinionated task author which is uh very opinionated right and is not something that models right and is not something that models right and is not something that models should have to adhere to. Um and finally should have to adhere to. Um and finally should have to adhere to. Um and finally uh leakage. So one thing about uh leakage. So one thing about uh leakage. So one thing about Sweetbench Pro is uh for very uh Sweetbench Pro is uh for very uh Sweetbench Pro is uh for very uh insightful models such as Claude, insightful models such as Claude, insightful models such as Claude, they're able to directly run git log and they're able to directly run git log and they're able to directly run git log and then go through the commit hashes and then go through the commit hashes and then go through the commit hashes and cherrypick the ones out that contain the cherrypick the ones out that contain the cherrypick the ones out that contain the golden patches which again very very golden patches which again very very golden patches which again very very serious issue. serious issue. serious issue. So this is deep suite. This is the So this is deep suite. This is the So this is deep suite. This is the updated leaderboard as of July 1st. You updated leaderboard as of July 1st. You updated leaderboard as of July 1st. You can see um I was mentioning before uh can see um I was mentioning before uh can see um I was mentioning before uh the problem of differentiating uh but the problem of differentiating uh but the problem of differentiating uh but you can see on deep suite here there is you can see on deep suite here there is you can see on deep suite here there is a very clear difference. there's a very a very clear difference. there's a very a very clear difference. there's a very clear performance gap between the top clear performance gap between the top clear performance gap between the top performing models versus you know at performing models versus you know at performing models versus you know at 10th place you have Gemini 3.1 Pro. Um 10th place you have Gemini 3.1 Pro. Um 10th place you have Gemini 3.1 Pro. Um also within the cloud and the GPT models also within the cloud and the GPT models also within the cloud and the GPT models as well we're able to see the some uh as well we're able to see the some uh as well we're able to see the some uh deviants and um yeah if you go on deviants and um yeah if you go on deviants and um yeah if you go on deepsweet.datacurve.ai deepsweet.datacurve.ai deepsweet.datacurve.ai you'll also be able to see um the token you'll also be able to see um the token you'll also be able to see um the token efficiency uh cost um token usage efficiency uh cost um token usage efficiency uh cost um token usage context window peak context all of that context window peak context all of that context window peak context all of that stuff on the deep site as well. But stuff on the deep site as well. But stuff on the deep site as well. But yeah, as of July 1st, uh, Fable 5 is yeah, as of July 1st, uh, Fable 5 is yeah, as of July 1st, uh, Fable 5 is retaining the top spot on our retaining the top spot on our retaining the top spot on our leaderboard.

leaderboard. leaderboard. So, um, the ranking information is So, um, the ranking information is So, um, the ranking information is available online. I wanted to talk about available online. I wanted to talk about available online. I wanted to talk about some of the qualitative insights into some of the qualitative insights into some of the qualitative insights into how these different models are how these different models are how these different models are performing, which I think is the most performing, which I think is the most performing, which I think is the most interesting part. Um, starting with the interesting part. Um, starting with the interesting part. Um, starting with the first one is we find claude is generally first one is we find claude is generally first one is we find claude is generally a very very um, thorough and exhaustive a very very um, thorough and exhaustive a very very um, thorough and exhaustive model. it will um try to explore model. it will um try to explore model. it will um try to explore everything including uh go through all everything including uh go through all everything including uh go through all of the git logs. Uh so one interesting of the git logs. Uh so one interesting of the git logs. Uh so one interesting insight was seeing that it becomes quite insight was seeing that it becomes quite insight was seeing that it becomes quite forgetful when it comes to multi-art forgetful when it comes to multi-art forgetful when it comes to multi-art prompts. So when you tell it uh within prompts. So when you tell it uh within prompts. So when you tell it uh within the scope of a task, let's say to in the scope of a task, let's say to in the scope of a task, let's say to in support both synchronous and async support both synchronous and async support both synchronous and async versions of calling a hook, it will go versions of calling a hook, it will go versions of calling a hook, it will go ahead and implement the synchronous ahead and implement the synchronous ahead and implement the synchronous part, but it may drop the asynchronous part, but it may drop the asynchronous part, but it may drop the asynchronous part. We observed this in roughly two part. We observed this in roughly two part. We observed this in roughly two out of three cloud rollouts across all out of three cloud rollouts across all out of three cloud rollouts across all of the uh trials, all of the rollouts of the uh trials, all of the rollouts of the uh trials, all of the rollouts that we ran. So this was definitely that we ran. So this was definitely that we ran. So this was definitely quite interesting because from my quite interesting because from my quite interesting because from my experiences and developers I've talked experiences and developers I've talked experiences and developers I've talked to as well, Claude is generally very to as well, Claude is generally very to as well, Claude is generally very very thorough thorough and able to get very thorough thorough and able to get very thorough thorough and able to get at the developer intent quite well.

at the developer intent quite well. at the developer intent quite well. Um another thing about claude is it pays Um another thing about claude is it pays Um another thing about claude is it pays very close attention to its environment. very close attention to its environment. very close attention to its environment. So it will often run uh this is taken So it will often run uh this is taken So it will often run uh this is taken from the trials uh we ran ourselves from the trials uh we ran ourselves from the trials uh we ran ourselves independently and also from examining independently and also from examining independently and also from examining sweetbench pro it'll attempt to run git sweetbench pro it'll attempt to run git sweetbench pro it'll attempt to run git log and recover the golden patch from log and recover the golden patch from log and recover the golden patch from the git history. We found that for opus the git history. We found that for opus the git history. We found that for opus 4.6 6 and 4.7 it did this 25% and 18% of 4.6 6 and 4.7 it did this 25% and 18% of 4.6 6 and 4.7 it did this 25% and 18% of the time respectively compared to all the time respectively compared to all the time respectively compared to all the Gemini models uh averaging at the Gemini models uh averaging at the Gemini models uh averaging at roughly 1% of the time and we found zero roughly 1% of the time and we found zero roughly 1% of the time and we found zero instances of this for the GPT models. So instances of this for the GPT models. So instances of this for the GPT models. So um thankfully within deep suite 1.1 we um thankfully within deep suite 1.1 we um thankfully within deep suite 1.1 we safeguarded further against models being safeguarded further against models being safeguarded further against models being able to cheat by pulling from the git able to cheat by pulling from the git able to cheat by pulling from the git history. Um but this was something yeah history. Um but this was something yeah history. Um but this was something yeah we observed quite frequently for claude we observed quite frequently for claude we observed quite frequently for claude within um the sweepbench pro rollouts. within um the sweepbench pro rollouts. within um the sweepbench pro rollouts. Um third finding is that GBT is very Um third finding is that GBT is very Um third finding is that GBT is very good at implementing exactly what it is good at implementing exactly what it is good at implementing exactly what it is asked across our failure mode analysis.

asked across our failure mode analysis. asked across our failure mode analysis. We found that it was the least likely We found that it was the least likely We found that it was the least likely model to miss requirements. Um GPT 5.4 model to miss requirements. Um GPT 5.4 model to miss requirements. Um GPT 5.4 was the second best model at this was the second best model at this was the second best model at this ranking only behind GPT 5.5. it always ranking only behind GPT 5.5. it always ranking only behind GPT 5.5. it always uh learns to read the prompts and theos uh learns to read the prompts and theos uh learns to read the prompts and theos rep repository contract very literally rep repository contract very literally rep repository contract very literally and producing a patch that honors the and producing a patch that honors the and producing a patch that honors the existing conventions um signatures existing conventions um signatures existing conventions um signatures within the repository which is very within the repository which is very within the repository which is very helpful and we found that these traits helpful and we found that these traits helpful and we found that these traits converge across all rollouts. So these converge across all rollouts. So these converge across all rollouts. So these were not just uh lucky attempts but on were not just uh lucky attempts but on were not just uh lucky attempts but on average like this was the uh favorable average like this was the uh favorable average like this was the uh favorable um behavior exhibited by GPT. um behavior exhibited by GPT. um behavior exhibited by GPT. Um and finally we found that on average Um and finally we found that on average Um and finally we found that on average stronger me stronger models have a great stronger me stronger models have a great stronger me stronger models have a great tendency to want to test their own work. tendency to want to test their own work. tendency to want to test their own work. Um but with a caveat in SweetBench Pro's Um but with a caveat in SweetBench Pro's Um but with a caveat in SweetBench Pro's template they explicitly tell the model template they explicitly tell the model template they explicitly tell the model that the tests are handled and therefore that the tests are handled and therefore that the tests are handled and therefore they do not need to uh write uh any new they do not need to uh write uh any new they do not need to uh write uh any new tests of their own. With that single tests of their own. With that single tests of their own. With that single line in the prompt it will uh prevent line in the prompt it will uh prevent line in the prompt it will uh prevent the models from even uh 5.5 and uh Opus the models from even uh 5.5 and uh Opus the models from even uh 5.5 and uh Opus 4.8 4.8 4.8 from attempting to verify its own work from attempting to verify its own work from attempting to verify its own work through the course of the rollout. Um in through the course of the rollout. Um in through the course of the rollout. Um in deep suite we do not have anything that deep suite we do not have anything that deep suite we do not have anything that says to write or to not to write tests says to write or to not to write tests says to write or to not to write tests and so we observe this uh divergence and so we observe this uh divergence and so we observe this uh divergence between uh the percentage of the time between uh the percentage of the time between uh the percentage of the time where these models are actually engaging where these models are actually engaging where these models are actually engaging and writing tests. So this is quite an and writing tests. So this is quite an and writing tests. So this is quite an important behavior as it can uh provide important behavior as it can uh provide important behavior as it can uh provide the models are trying to obtain their the models are trying to obtain their the models are trying to obtain their own ways to verify and validate their own ways to verify and validate their own ways to verify and validate their work through the course of a rollout. uh

work through the course of a rollout. uh work through the course of a rollout. uh we find on average that stronger models we find on average that stronger models we find on average that stronger models like 5.4 4.7 exhibit this the majority like 5.4 4.7 exhibit this the majority like 5.4 4.7 exhibit this the majority of the time whereas uh models like three of the time whereas uh models like three of the time whereas uh models like three flash and 3.1 pro are far less flash and 3.1 pro are far less flash and 3.1 pro are far less frequently um willing to test their own frequently um willing to test their own frequently um willing to test their own work. Um yeah so uh takeaways from the work. Um yeah so uh takeaways from the work. Um yeah so uh takeaways from the findings I think is is very interesting findings I think is is very interesting findings I think is is very interesting how um yeah stronger models on average how um yeah stronger models on average how um yeah stronger models on average exhibit um or converge on these exhibit um or converge on these exhibit um or converge on these behaviors. Um so moving on to the tasks behaviors. Um so moving on to the tasks behaviors. Um so moving on to the tasks right the the methodology behind Deep right the the methodology behind Deep right the the methodology behind Deep Suite um we made a decision to want to Suite um we made a decision to want to Suite um we made a decision to want to have every task authored uh from scratch have every task authored uh from scratch have every task authored uh from scratch uh rather than being mined. Um aside uh rather than being mined. Um aside uh rather than being mined. Um aside from the uh issues with contamination from the uh issues with contamination from the uh issues with contamination that we mentioned previously. Um this that we mentioned previously. Um this that we mentioned previously. Um this also plays into uh one of our core also plays into uh one of our core also plays into uh one of our core strengths which is that we offer a strengths which is that we offer a strengths which is that we offer a bespoke platform where we have um bespoke platform where we have um bespoke platform where we have um software engineers, machine learning software engineers, machine learning software engineers, machine learning enthusiasts come on and create these enthusiasts come on and create these enthusiasts come on and create these challenges and compete against one challenges and compete against one challenges and compete against one another. Um this platform is like another. Um this platform is like another. Um this platform is like shipped and or sorry this platform is shipped and or sorry this platform is shipped and or sorry this platform is called shipped and we have a version of called shipped and we have a version of called shipped and we have a version of this platform for every single domain this platform for every single domain this platform for every single domain that we're interested in. For example, that we're interested in. For example, that we're interested in. For example, for software engineering, it takes a lot for software engineering, it takes a lot for software engineering, it takes a lot after code forces or GitHub. Um, and after code forces or GitHub. Um, and after code forces or GitHub. Um, and we're really looking for enthusiasts.

we're really looking for enthusiasts. we're really looking for enthusiasts. So, these are oftent times open-source So, these are oftent times open-source So, these are oftent times open-source engineers who are core contributors or engineers who are core contributors or engineers who are core contributors or maintainers of the projects that they're maintainers of the projects that they're maintainers of the projects that they're actively making tasks for. So, by actively making tasks for. So, by actively making tasks for. So, by creating these tasks from scratch, we creating these tasks from scratch, we creating these tasks from scratch, we know that the outputs are intrinsically know that the outputs are intrinsically know that the outputs are intrinsically aligned with our objective of providing aligned with our objective of providing aligned with our objective of providing a fair and comprehensive test to models. a fair and comprehensive test to models. a fair and comprehensive test to models. We also know that these people have have We also know that these people have have We also know that these people have have very thorough understandings of the very thorough understandings of the very thorough understandings of the repositories philosophy and the existing repositories philosophy and the existing repositories philosophy and the existing conventions. So they can make tasks that conventions. So they can make tasks that conventions. So they can make tasks that are both realistic in terms of the are both realistic in terms of the are both realistic in terms of the prompt but also realistic in the sense prompt but also realistic in the sense prompt but also realistic in the sense that uh this is an actual PR that you that uh this is an actual PR that you that uh this is an actual PR that you might see getting merged into the might see getting merged into the might see getting merged into the repositories. Um another uh very important design Um another uh very important design decision is we try as much as possible decision is we try as much as possible decision is we try as much as possible to make our prompts read like real to make our prompts read like real to make our prompts read like real tasks. On average, the average prompt uh tasks. On average, the average prompt uh tasks. On average, the average prompt uh characters within SweetBench Pro is over characters within SweetBench Pro is over characters within SweetBench Pro is over 4,500 characters, whereas for us, it's 4,500 characters, whereas for us, it's 4,500 characters, whereas for us, it's uh roughly half of that. And it's this uh roughly half of that. And it's this uh roughly half of that. And it's this is important because when you're is important because when you're is important because when you're prompting say a junior engineer or prompting say a junior engineer or prompting say a junior engineer or you're prompting a model to solve a very you're prompting a model to solve a very you're prompting a model to solve a very uh high ceiling ambiguous task, you're uh high ceiling ambiguous task, you're uh high ceiling ambiguous task, you're not going to be coming in there with a not going to be coming in there with a not going to be coming in there with a to-do list uh telling it to oh first do to-do list uh telling it to oh first do to-do list uh telling it to oh first do this and then do this and then write this and then do this and then write this and then do this and then write this function signature in exactly uh this function signature in exactly uh this function signature in exactly uh this way that I've prescribed on to you.

this way that I've prescribed on to you. this way that I've prescribed on to you. oftentimes you're going to give it the oftentimes you're going to give it the oftentimes you're going to give it the high level objective, get it to explore high level objective, get it to explore high level objective, get it to explore and get it to reason about to uh the and get it to reason about to uh the and get it to reason about to uh the list of to-dos and ultimately to the list of to-dos and ultimately to the list of to-dos and ultimately to the solution on its own. Um so this was not solution on its own. Um so this was not solution on its own. Um so this was not the case in SweetBench Pro. It's very the case in SweetBench Pro. It's very the case in SweetBench Pro. It's very overly verbose and trying to prescribe a overly verbose and trying to prescribe a overly verbose and trying to prescribe a certain solution method onto agents. Um, certain solution method onto agents. Um, certain solution method onto agents. Um, as much as we could, we try and make as much as we could, we try and make as much as we could, we try and make Deep Suite prompts as tur and as uh high Deep Suite prompts as tur and as uh high Deep Suite prompts as tur and as uh high level as possible. Um, mirroring what level as possible. Um, mirroring what level as possible. Um, mirroring what you might see in the real world if you you might see in the real world if you you might see in the real world if you were to prompt, say, another another were to prompt, say, another another were to prompt, say, another another engineer or one of your agents to go and engineer or one of your agents to go and engineer or one of your agents to go and solve an engineering task. solve an engineering task. solve an engineering task. Um, so even though our prompts are Um, so even though our prompts are Um, so even though our prompts are short, we still are able to maintain the short, we still are able to maintain the short, we still are able to maintain the long horizon nature of these tasks. Even long horizon nature of these tasks. Even long horizon nature of these tasks. Even with our prompts again being roughly with our prompts again being roughly with our prompts again being roughly half the size of Sweetbench Pros, we half the size of Sweetbench Pros, we half the size of Sweetbench Pros, we find that the average size of our find that the average size of our find that the average size of our solution is five times the lines of code solution is five times the lines of code solution is five times the lines of code um compared to Sweepbench Pros. Um we um compared to Sweepbench Pros. Um we um compared to Sweepbench Pros. Um we also verified that uh there are on also verified that uh there are on also verified that uh there are on average seven files being touched in the average seven files being touched in the average seven files being touched in the agents solution. Um and across the agents solution. Um and across the agents solution. Um and across the course of a roll out we have two times course of a roll out we have two times course of a roll out we have two times more output tokens being emitted. Um more output tokens being emitted. Um more output tokens being emitted. Um and um finally we have uh we verifier and um finally we have uh we verifier and um finally we have uh we verifier design is of course one of the most um design is of course one of the most um design is of course one of the most um important and tricky parts of building important and tricky parts of building important and tricky parts of building good environments. In Sweepbench Pro we good environments. In Sweepbench Pro we good environments. In Sweepbench Pro we have these verifiers that are testing have these verifiers that are testing have these verifiers that are testing again for specific implementations. It again for specific implementations. It again for specific implementations. It will fail the model if it uh produces a will fail the model if it uh produces a will fail the model if it uh produces a function that may address the objective function that may address the objective function that may address the objective but is not named or is not defined but is not named or is not defined but is not named or is not defined within a specific module or if there is

within a specific module or if there is within a specific module or if there is the absence of specific helpers or other the absence of specific helpers or other the absence of specific helpers or other private functions. Um because again private functions. Um because again private functions. Um because again these are derived from the solutions these are derived from the solutions these are derived from the solutions that were merged in the actual PR. Um so that were merged in the actual PR. Um so that were merged in the actual PR. Um so for us we want to uh emphasize on the for us we want to uh emphasize on the for us we want to uh emphasize on the observable behavior as much as possible. observable behavior as much as possible. observable behavior as much as possible. We want to ensure that any correct We want to ensure that any correct We want to ensure that any correct implementation uh anything that implementation uh anything that implementation uh anything that correctly solves the problem is rewarded correctly solves the problem is rewarded correctly solves the problem is rewarded and this will prevent against uh false and this will prevent against uh false and this will prevent against uh false negatives. We also make sure that um negatives. We also make sure that um negatives. We also make sure that um there's the absence of these PR derived there's the absence of these PR derived there's the absence of these PR derived tests that rely on naming relying on tests that rely on naming relying on tests that rely on naming relying on specific implementations. Um and so this specific implementations. Um and so this specific implementations. Um and so this will prevent again again uh again uh will prevent again again uh again uh will prevent again again uh again uh towards false negatives as well. And we towards false negatives as well. And we towards false negatives as well. And we observed through a combination of these observed through a combination of these observed through a combination of these um you know considerations we're able to um you know considerations we're able to um you know considerations we're able to drastically reduce the false negative as drastically reduce the false negative as drastically reduce the false negative as well as the false positive rates when we well as the false positive rates when we well as the false positive rates when we analyzed um our rollouts compared to analyzed um our rollouts compared to analyzed um our rollouts compared to SweetBench pros uh using both human SweetBench pros uh using both human SweetBench pros uh using both human experts as well as LLM as judge and yes the coverage for us spans across and yes the coverage for us spans across these 91 repositories. Um our criteria these 91 repositories. Um our criteria these 91 repositories. Um our criteria for these repositories was uh ones that for these repositories was uh ones that for these repositories was uh ones that had more than 500 stars on GitHub. Uh had more than 500 stars on GitHub. Uh had more than 500 stars on GitHub. Uh they are actively being contributed they are actively being contributed they are actively being contributed towards and for for our pool of uh uh towards and for for our pool of uh uh towards and for for our pool of uh uh subject matter experts to validate that subject matter experts to validate that subject matter experts to validate that these are you know in fact repositories these are you know in fact repositories these are you know in fact repositories that are actively used in in in the in that are actively used in in in the in that are actively used in in in the in the real world and are representative the real world and are representative the real world and are representative and can field uh real world uh and and can field uh real world uh and and can field uh real world uh and realistic software engineering tasks.

realistic software engineering tasks. realistic software engineering tasks. Um but with all that said um there's Um but with all that said um there's Um but with all that said um there's still a lot of work to be done for deep still a lot of work to be done for deep still a lot of work to be done for deep suite and for benchmarks in general. One suite and for benchmarks in general. One suite and for benchmarks in general. One of the things that we outlined in our of the things that we outlined in our of the things that we outlined in our blog is our uh choice to use mini suite blog is our uh choice to use mini suite blog is our uh choice to use mini suite agent which is an agent agnostic agent which is an agent agnostic agent which is an agent agnostic harness. The reason why here is we harness. The reason why here is we harness. The reason why here is we really want to be focusing on the really want to be focusing on the really want to be focusing on the model's base performance and so we use model's base performance and so we use model's base performance and so we use mini suite agent also uh ran rollouts to mini suite agent also uh ran rollouts to mini suite agent also uh ran rollouts to test that the uh performance is test that the uh performance is test that the uh performance is comparable both using mini suite and comparable both using mini suite and comparable both using mini suite and against each model's native harness. But against each model's native harness. But against each model's native harness. But I think there's a lot of work to be done I think there's a lot of work to be done I think there's a lot of work to be done in the future for benchmarks that focus in the future for benchmarks that focus in the future for benchmarks that focus solely or more so on harnesses and solely or more so on harnesses and solely or more so on harnesses and comparing the effects that these comparing the effects that these comparing the effects that these harnesses whether it's native or third harnesses whether it's native or third harnesses whether it's native or third party ones like mini suite um towards party ones like mini suite um towards party ones like mini suite um towards the efficiency and the output of these the efficiency and the output of these the efficiency and the output of these models. Another thing we want to improve models. Another thing we want to improve models. Another thing we want to improve on is task mix. So given that we are on is task mix. So given that we are on is task mix. So given that we are targeting long horizon tasks naturally targeting long horizon tasks naturally targeting long horizon tasks naturally this meant that there's less um less uh this meant that there's less um less uh this meant that there's less um less uh emphasis on bug localization and emphasis on bug localization and emphasis on bug localization and refactoring these are obviously very refactoring these are obviously very refactoring these are obviously very representative of real work that representative of real work that representative of real work that software engineers are doing uh under software engineers are doing uh under software engineers are doing uh under reppresented in our current taxonomy uh reppresented in our current taxonomy uh reppresented in our current taxonomy uh for deep suite. Um and finally, for deep suite. Um and finally, for deep suite. Um and finally, repository pool. Um we put an emphasis repository pool. Um we put an emphasis repository pool. Um we put an emphasis on trying to field as many diverse on trying to field as many diverse on trying to field as many diverse repositories as possible, keeping the repositories as possible, keeping the repositories as possible, keeping the median tasks per repository to a very median tasks per repository to a very median tasks per repository to a very low count. But uh further work here just low count. But uh further work here just low count. But uh further work here just to pull in more or more repos, more to pull in more or more repos, more to pull in more or more repos, more tasks that software engineers find tasks that software engineers find tasks that software engineers find interesting and find them to be good and interesting and find them to be good and interesting and find them to be good and maybe also um more niche t tests of

maybe also um more niche t tests of maybe also um more niche t tests of models performance would also be a great models performance would also be a great models performance would also be a great addition here. addition here. addition here. So we've already uh released deep suite So we've already uh released deep suite So we've already uh released deep suite v 1.1. So in here we've taken some v 1.1. So in here we've taken some v 1.1. So in here we've taken some additional measures to guard against additional measures to guard against additional measures to guard against cheating uh reward hacking uh by cheating uh reward hacking uh by cheating uh reward hacking uh by ensuring you know the verifier runtime ensuring you know the verifier runtime ensuring you know the verifier runtime is fully separate now from the agent is fully separate now from the agent is fully separate now from the agent runtime. Um also making sure the test runtime. Um also making sure the test runtime. Um also making sure the test reports are in a more standardized reports are in a more standardized reports are in a more standardized format and also making sure that we've format and also making sure that we've format and also making sure that we've trimmed all of the git refs and the trimmed all of the git refs and the trimmed all of the git refs and the commits besides the uh base commit that commits besides the uh base commit that commits besides the uh base commit that our agents are working on. So all of our agents are working on. So all of our agents are working on. So all of this in service of just making the this in service of just making the this in service of just making the environments more robust and more environments more robust and more environments more robust and more cheating proof. But uh as I mentioned cheating proof. But uh as I mentioned cheating proof. But uh as I mentioned looking ahead we want to support an even looking ahead we want to support an even looking ahead we want to support an even greater diversity of like t uh tasks greater diversity of like t uh tasks greater diversity of like t uh tasks corpus. We also want to look into hybrid corpus. We also want to look into hybrid corpus. We also want to look into hybrid verification because if we're able to verification because if we're able to verification because if we're able to use LLM as judge or other um use LLM as judge or other um use LLM as judge or other um methodologies, it's possible for us to methodologies, it's possible for us to methodologies, it's possible for us to make our prompts even more tur and even make our prompts even more tur and even make our prompts even more tur and even more um even more uh high level and more um even more uh high level and more um even more uh high level and focus on the objective rather than focus on the objective rather than focus on the objective rather than prescribing anything onto the agent. Um prescribing anything onto the agent. Um prescribing anything onto the agent. Um there is of course like a certain degree there is of course like a certain degree there is of course like a certain degree that we have to in our current prompts that we have to in our current prompts that we have to in our current prompts like um hint the agents steering them like um hint the agents steering them like um hint the agents steering them towards a current methodology just towards a current methodology just towards a current methodology just because otherwise they they may not be because otherwise they they may not be because otherwise they they may not be well positioned at all to make well positioned at all to make well positioned at all to make meaningful progress towards the task. Um meaningful progress towards the task. Um meaningful progress towards the task. Um but something like LLM as a judges and but something like LLM as a judges and but something like LLM as a judges and hybrid verifiers would potentially help hybrid verifiers would potentially help hybrid verifiers would potentially help us towards that and beyond deep suite us towards that and beyond deep suite us towards that and beyond deep suite we're also working on new benchmarks we're also working on new benchmarks we're also working on new benchmarks that are in the works. These are again

that are in the works. These are again that are in the works. These are again focused on the highv value domains that focused on the highv value domains that focused on the highv value domains that uh data curve we prioritize uh as being uh data curve we prioritize uh as being uh data curve we prioritize uh as being the domains where we want to be uh most the domains where we want to be uh most the domains where we want to be uh most meaningfully advancing model meaningfully advancing model meaningfully advancing model capabilities. capabilities. capabilities. Um but with that said we're actively Um but with that said we're actively Um but with that said we're actively hiring uh both researchers, engineers hiring uh both researchers, engineers hiring uh both researchers, engineers helping us with these new benchmarks, helping us with these new benchmarks, helping us with these new benchmarks, new trading data pipelines in service of new trading data pipelines in service of new trading data pipelines in service of advancing these capabilities. So um advancing these capabilities. So um advancing these capabilities. So um definitely reach out at definitely reach out at definitely reach out at datacurve.ai/careers. datacurve.ai/careers. datacurve.ai/careers. Um, and yeah, if you're interested about Um, and yeah, if you're interested about Um, and yeah, if you're interested about any of this research benchmark or any of any of this research benchmark or any of any of this research benchmark or any of our works, uh, come find me after. Thank our works, uh, come find me after. Thank our works, uh, come find me after. Thank you very much. [applause]