Verifiable Environments for AI in Biology — Kenny Workman, LatchBio

Thank you to the organizers for having Thank you to the organizers for having me. I'm one of the co-founders and CTO me. I'm one of the co-founders and CTO me. I'm one of the co-founders and CTO at Latch. at Latch. at Latch. We are basically a vertical AI lab for We are basically a vertical AI lab for We are basically a vertical AI lab for benchmark and agent engineering. I'm benchmark and agent engineering. I'm benchmark and agent engineering. I'm hoping to motivate and explain exactly hoping to motivate and explain exactly hoping to motivate and explain exactly what that means today. what that means today. what that means today. Starting directly with motivation for Starting directly with motivation for Starting directly with motivation for agents in in bio generally. agents in in bio generally. agents in in bio generally. Um Many people in my domain are familiar Um Many people in my domain are familiar Um Many people in my domain are familiar with this curve, but this is basically with this curve, but this is basically with this curve, but this is basically the log linear curve of data generated the log linear curve of data generated the log linear curve of data generated over the years in in biology. And the over the years in in biology. And the over the years in in biology. And the reason I'm bringing it up it will become reason I'm bringing it up it will become reason I'm bringing it up it will become directly important to the kinds of directly important to the kinds of directly important to the kinds of things we want to do in engineering. Um things we want to do in engineering. Um things we want to do in engineering. Um This curve is driven by a very small This curve is driven by a very small This curve is driven by a very small handful of experimental classes. handful of experimental classes. handful of experimental classes. One is called single cell biology. This One is called single cell biology. This One is called single cell biology. This is where we split up cells, break them is where we split up cells, break them is where we split up cells, break them apart, and measure their RNA. apart, and measure their RNA. apart, and measure their RNA. The second is spatial biology, which The second is spatial biology, which The second is spatial biology, which will become the focus of the next will become the focus of the next will become the focus of the next segment of the talk. Same thing as segment of the talk. Same thing as segment of the talk. Same thing as single cell, but you get spatial single cell, but you get spatial single cell, but you get spatial resolution. You can look at how RNA is resolution. You can look at how RNA is resolution. You can look at how RNA is spread out geometrically over a tissue. spread out geometrically over a tissue. spread out geometrically over a tissue. And the third thing is proteomics. It's And the third thing is proteomics. It's And the third thing is proteomics. It's a broad category of different a broad category of different a broad category of different techniques. They measure proteins. Um techniques. They measure proteins. Um techniques. They measure proteins. Um Less less abundant in ordering. Like Less less abundant in ordering. Like Less less abundant in ordering. Like less less data volume generated relative less less data volume generated relative less less data volume generated relative to the other two, but still important.

to the other two, but still important. to the other two, but still important. Um you guys are technical and I always Um you guys are technical and I always Um you guys are technical and I always think it's good to ground things think it's good to ground things think it's good to ground things somewhat quantitatively, but um these somewhat quantitatively, but um these somewhat quantitatively, but um these are really big numbers and the are really big numbers and the are really big numbers and the experimental data from these techniques experimental data from these techniques experimental data from these techniques is growing quite rapidly. is growing quite rapidly. is growing quite rapidly. Um Um Um Almost greater than any other domain of Almost greater than any other domain of Almost greater than any other domain of science other than particle collider um science other than particle collider um science other than particle collider um machines. Single cell experiments can machines. Single cell experiments can machines. Single cell experiments can yield two to six terabytes per run. yield two to six terabytes per run. yield two to six terabytes per run. Spatial runs can yield seven terabytes Spatial runs can yield seven terabytes Spatial runs can yield seven terabytes of run. Proteomics a few hundred gigs. of run. Proteomics a few hundred gigs. of run. Proteomics a few hundred gigs. Um and the only reason I bring this up Um and the only reason I bring this up Um and the only reason I bring this up is to say hey like is to say hey like is to say hey like the output of a single experiment can the output of a single experiment can the output of a single experiment can exceed what a scientist can safely store exceed what a scientist can safely store exceed what a scientist can safely store on a consumer laptop in many cases. on a consumer laptop in many cases. on a consumer laptop in many cases. And the laws driving how the the And the laws driving how the the And the laws driving how the the molecular capture works point to rapid molecular capture works point to rapid molecular capture works point to rapid gains in this throughput over the coming gains in this throughput over the coming gains in this throughput over the coming years. years. years. Um one thing I like to do uh is when I Um one thing I like to do uh is when I Um one thing I like to do uh is when I read a new paper is decompose it and read a new paper is decompose it and read a new paper is decompose it and align it to this framework because it align it to this framework because it align it to this framework because it will be important in a second. Uh modern will be important in a second. Uh modern will be important in a second. Uh modern biology research is centered around biology research is centered around biology research is centered around those experiments. those experiments. those experiments. You basically choose a model You basically choose a model You basically choose a model biological model, not the kind of models biological model, not the kind of models biological model, not the kind of models you guys are used to. You generate data you guys are used to. You generate data you guys are used to. You generate data from that model, you process the data, from that model, you process the data, from that model, you process the data, you creatively think about the results you creatively think about the results you creatively think about the results in the context of prior literature, and in the context of prior literature, and in the context of prior literature, and you make a claim.

you make a claim. you make a claim. Almost all modern experiments papers Almost all modern experiments papers Almost all modern experiments papers that you see published follow this loose that you see published follow this loose that you see published follow this loose structure with a lot of nuance. Um structure with a lot of nuance. Um structure with a lot of nuance. Um all all all that to say is they be they all all all that to say is they be they all all all that to say is they be they become something of a a panning become something of a a panning become something of a a panning experiment. You're looking for a signal experiment. You're looking for a signal experiment. You're looking for a signal using measurement in a sea of noise. using measurement in a sea of noise. using measurement in a sea of noise. Um and so this is building up to the Um and so this is building up to the Um and so this is building up to the claim that like code and sweet data claim that like code and sweet data claim that like code and sweet data analysis scaffold to genetic biology. It analysis scaffold to genetic biology. It analysis scaffold to genetic biology. It becomes this executable substrate that becomes this executable substrate that becomes this executable substrate that we can use to train things. It we can use to train things. It we can use to train things. It introduces a natural way to benchmark introduces a natural way to benchmark introduces a natural way to benchmark and climb capability. Um I've written and climb capability. Um I've written and climb capability. Um I've written about this a lot at this blog. Uh link about this a lot at this blog. Uh link about this a lot at this blog. Uh link here. here. here. There's a lot more depth to this claim, There's a lot more depth to this claim, There's a lot more depth to this claim, so I wouldn't take it at face value, but so I wouldn't take it at face value, but so I wouldn't take it at face value, but it's something to look into. All you can it's something to look into. All you can it's something to look into. All you can take away from this is like just like take away from this is like just like take away from this is like just like code provided a verifiable substrate for code provided a verifiable substrate for code provided a verifiable substrate for complex software tasks that are not complex software tasks that are not complex software tasks that are not inherently verifiable, uh data analysis inherently verifiable, uh data analysis inherently verifiable, uh data analysis might do the same thing in bio. might do the same thing in bio. might do the same thing in bio. So how do we get started? We were So how do we get started? We were So how do we get started? We were originally originally originally a data tool vendor for biotech and a data tool vendor for biotech and a data tool vendor for biotech and pharma. pharma. pharma. Um where we started 5 years ago out of Um where we started 5 years ago out of Um where we started 5 years ago out of Berkeley on Berkeley on Berkeley on 25. We started when I was 20. We stored, 25. We started when I was 20. We stored, 25. We started when I was 20. We stored, transformed, filed data from large transformed, filed data from large transformed, filed data from large experiments as a service. Uh we'll try experiments as a service. Uh we'll try experiments as a service. Uh we'll try to build products, explored lots of to build products, explored lots of to build products, explored lots of things.

things. things. Over the last 2 years, um Over the last 2 years, um Over the last 2 years, um we started moving away from biotech and we started moving away from biotech and we started moving away from biotech and pharma and more towards the people who pharma and more towards the people who pharma and more towards the people who build those kits I was talking about. build those kits I was talking about. build those kits I was talking about. Package the software into kind of Package the software into kind of Package the software into kind of white-labeled things that they s- white-labeled things that they s- white-labeled things that they s- provided to the scientists themselves. provided to the scientists themselves. provided to the scientists themselves. Um Um Um Help them analyze their data. And then Help them analyze their data. And then Help them analyze their data. And then over time, there became this strong over time, there became this strong over time, there became this strong interaction uh with the agents uh interaction uh with the agents uh interaction uh with the agents uh using the infrastructure components as using the infrastructure components as using the infrastructure components as tools and the loop context you guys are tools and the loop context you guys are tools and the loop context you guys are familiar with. Except in our domain, the familiar with. Except in our domain, the familiar with. Except in our domain, the tools can take days or weeks. I'm tools can take days or weeks. I'm tools can take days or weeks. I'm serious. serious. serious. Um Um Um What started to happen around last What started to happen around last What started to happen around last summer is agent prototypes uh started to summer is agent prototypes uh started to summer is agent prototypes uh started to work. So, we took uh coding models uh to work. So, we took uh coding models uh to work. So, we took uh coding models uh to our knowledge at the time they were not our knowledge at the time they were not our knowledge at the time they were not seriously post-trained on any tasks in seriously post-trained on any tasks in seriously post-trained on any tasks in biology to this point. And we started to biology to this point. And we started to biology to this point. And we started to build products that look a lot like all build products that look a lot like all build products that look a lot like all the other agent products. the other agent products. the other agent products. They have a chat interface for you to They have a chat interface for you to They have a chat interface for you to ask questions to and they build ask questions to and they build ask questions to and they build dashboards and dispatch operations to dashboards and dispatch operations to dashboards and dispatch operations to external compute. Um the kinds of things external compute. Um the kinds of things external compute. Um the kinds of things these that this this agent would do is these that this this agent would do is these that this this agent would do is take large file data from the types of take large file data from the types of take large file data from the types of experiments I was talking about earlier.

experiments I was talking about earlier. experiments I was talking about earlier. Say like uh Say like uh Say like uh tissue bisect biopsy from cancer with a tissue bisect biopsy from cancer with a tissue bisect biopsy from cancer with a spatial measurement. And then the the spatial measurement. And then the the spatial measurement. And then the the scientist is just iterating with it to scientist is just iterating with it to scientist is just iterating with it to get at some question they have. You get at some question they have. You get at some question they have. You know, maybe between a malignant and know, maybe between a malignant and know, maybe between a malignant and non-malignant part non-malignant part non-malignant part of the tissue, what kind of genes are of the tissue, what kind of genes are of the tissue, what kind of genes are being overexpressed? being overexpressed? being overexpressed? Um but what was fascinating is Um but what was fascinating is Um but what was fascinating is even though it was pretty bad, it showed even though it was pretty bad, it showed even though it was pretty bad, it showed the early signs of working. And uh the early signs of working. And uh the early signs of working. And uh It became clear to us at this time that It became clear to us at this time that It became clear to us at this time that agentic biology might look a lot like agentic biology might look a lot like agentic biology might look a lot like code. Actually lifted this slide from code. Actually lifted this slide from code. Actually lifted this slide from Anthropic's Anthropic's Anthropic's um Claude science announcement um Claude science announcement um Claude science announcement yesterday, but yesterday, but yesterday, but we uh just like, you know, you had this we uh just like, you know, you had this we uh just like, you know, you had this like kind of faulty like kind of faulty like kind of faulty silly engineer that became better and silly engineer that became better and silly engineer that became better and then as it improved in capability, you then as it improved in capability, you then as it improved in capability, you could dispatch work to teams of them could dispatch work to teams of them could dispatch work to teams of them that work together. The same pattern that work together. The same pattern that work together. The same pattern will probably emerge in science. will probably emerge in science. will probably emerge in science. And products and harnesses will emerge And products and harnesses will emerge And products and harnesses will emerge to orchestrate uh work and abstract it to orchestrate uh work and abstract it to orchestrate uh work and abstract it so that teams of teams of agentic so that teams of teams of agentic so that teams of teams of agentic scientists can take on capability. But scientists can take on capability. But scientists can take on capability. But we needed focused post-training. we needed focused post-training. we needed focused post-training. Uh cuz at the time and still now, um Uh cuz at the time and still now, um Uh cuz at the time and still now, um frontier models cannot be trusted to do frontier models cannot be trusted to do frontier models cannot be trusted to do real work. They're missing some real work. They're missing some real work. They're missing some capability between knowing biology and capability between knowing biology and capability between knowing biology and writing code. And this is exactly writing code. And this is exactly writing code. And this is exactly extracting scientific insight from extracting scientific insight from extracting scientific insight from real-world data.

real-world data. real-world data. Unlike code, it which is one constituent Unlike code, it which is one constituent Unlike code, it which is one constituent component of this work, uh it also component of this work, uh it also component of this work, uh it also involves data analysis and domain involves data analysis and domain involves data analysis and domain reasoning, scientific reasoning. reasoning, scientific reasoning. reasoning, scientific reasoning. We thought spatial biology is a good We thought spatial biology is a good We thought spatial biology is a good place to start, so we started building place to start, so we started building place to start, so we started building agents. This is a technical green field. agents. This is a technical green field. agents. This is a technical green field. We had many existing customers. We had many existing customers. We had many existing customers. Uh it's also just a beautiful example of Uh it's also just a beautiful example of Uh it's also just a beautiful example of measurement drives progress. You can measurement drives progress. You can measurement drives progress. You can actually see biological phenomena play actually see biological phenomena play actually see biological phenomena play out. out. out. Um you can look at a developing mouse Um you can look at a developing mouse Um you can look at a developing mouse embryo. embryo. embryo. Um Um Um And we had to get in the guts of how the And we had to get in the guts of how the And we had to get in the guts of how the data was captured and analyzed to build data was captured and analyzed to build data was captured and analyzed to build good agents. good agents. good agents. Um I'm not going to get into this in Um I'm not going to get into this in Um I'm not going to get into this in detail, but I put up this tree of uh detail, but I put up this tree of uh detail, but I put up this tree of uh different capture technologies in different capture technologies in different capture technologies in spatial biology to highlight the spatial biology to highlight the spatial biology to highlight the diversity of things that exist. They diversity of things that exist. They diversity of things that exist. They really span advances in chemistry, really span advances in chemistry, really span advances in chemistry, optics, semiconductors, physics. optics, semiconductors, physics. optics, semiconductors, physics. Each branch uh is induced from decades Each branch uh is induced from decades Each branch uh is induced from decades of cumulative work to figure out how to of cumulative work to figure out how to of cumulative work to figure out how to measure a a type of molecule. As a measure a a type of molecule. As a measure a a type of molecule. As a specific example, one technique we work specific example, one technique we work specific example, one technique we work with is called sequencing base spatial, with is called sequencing base spatial, with is called sequencing base spatial, and it's where you take a slide of and it's where you take a slide of and it's where you take a slide of little beads with clumps of DNA attached little beads with clumps of DNA attached little beads with clumps of DNA attached to them that fuse to the RNA inside of a to them that fuse to the RNA inside of a to them that fuse to the RNA inside of a tissue section, so tissue section, so tissue section, so biologists can like lay like a chunk of biologists can like lay like a chunk of biologists can like lay like a chunk of a tumor over it, and then it'll capture a tumor over it, and then it'll capture a tumor over it, and then it'll capture all the RNA in it, and then let you know all the RNA in it, and then let you know all the RNA in it, and then let you know with precise uh geometric resolution with precise uh geometric resolution with precise uh geometric resolution where the RNA was.

where the RNA was. where the RNA was. It's cool stuff. The data, when it comes It's cool stuff. The data, when it comes It's cool stuff. The data, when it comes out, ends up looking like a big matrix out, ends up looking like a big matrix out, ends up looking like a big matrix of numbers in a large high-content of numbers in a large high-content of numbers in a large high-content image. Um you have to take it through a image. Um you have to take it through a image. Um you have to take it through a sequence of steps sequence of steps sequence of steps uh to get to the end thing that you uh to get to the end thing that you uh to get to the end thing that you want. These steps are highly variable, want. These steps are highly variable, want. These steps are highly variable, especially across technology types, especially across technology types, especially across technology types, tissue, disease contexts. Um there isn't tissue, disease contexts. Um there isn't tissue, disease contexts. Um there isn't a lot of consensus in the field a lot of consensus in the field a lot of consensus in the field uh for each step, so we really needed a uh for each step, so we really needed a uh for each step, so we really needed a measuring stick to understand if the measuring stick to understand if the measuring stick to understand if the agents we're building were doing agents we're building were doing agents we're building were doing scientific work. scientific work. scientific work. Uh the existing benchmarks we saw at the Uh the existing benchmarks we saw at the Uh the existing benchmarks we saw at the time did not measure the tasks relevant time did not measure the tasks relevant time did not measure the tasks relevant to this category of work. Um they mostly to this category of work. Um they mostly to this category of work. Um they mostly measured things in a Q&A settings, like measured things in a Q&A settings, like measured things in a Q&A settings, like what what do in a kind of academic way, what what do in a kind of academic way, what what do in a kind of academic way, or they weren't sufficiently focused on or they weren't sufficiently focused on or they weren't sufficiently focused on the experiment type. This is an actual the experiment type. This is an actual the experiment type. This is an actual screenshot of screenshot of screenshot of Anthropic's model card at the time that Anthropic's model card at the time that Anthropic's model card at the time that we built this benchmark. So, we built we built this benchmark. So, we built we built this benchmark. So, we built one. one. one. It's called spatial bench. Um last It's called spatial bench. Um last It's called spatial bench. Um last December, there's 146 problems. They December, there's 146 problems. They December, there's 146 problems. They spanned the different kits I talked spanned the different kits I talked spanned the different kits I talked about or attempted to. about or attempted to. about or attempted to. And then they spanned all those And then they spanned all those And then they spanned all those different tasks that I talked about as different tasks that I talked about as different tasks that I talked about as well. well. well. So, the thing that we found at this So, the thing that we found at this So, the thing that we found at this time, and still to an extent is true time, and still to an extent is true time, and still to an extent is true today, is the the grading of these end today, is the the grading of these end today, is the the grading of these end outcomes in biology uh is outcomes in biology uh is outcomes in biology uh is too sparse because the models are pretty too sparse because the models are pretty too sparse because the models are pretty bad. So, you have to break things up bad. So, you have to break things up bad. So, you have to break things up into manageable chunks to get some into manageable chunks to get some into manageable chunks to get some semblance of verifiability.

semblance of verifiability. semblance of verifiability. And that's kind of induced by um And that's kind of induced by um And that's kind of induced by um sticking to these little components of sticking to these little components of sticking to these little components of like that DAG, that analysis DAG. Um like that DAG, that analysis DAG. Um like that DAG, that analysis DAG. Um getting data to a state where it would getting data to a state where it would getting data to a state where it would exist right before a scientist or exist right before a scientist or exist right before a scientist or theoretician could do work on it, and theoretician could do work on it, and theoretician could do work on it, and then then then um figuring out what the ground truth um figuring out what the ground truth um figuring out what the ground truth would be in that context. So, a single would be in that context. So, a single would be in that context. So, a single evaluation kind of looks like one or evaluation kind of looks like one or evaluation kind of looks like one or more data nodes, again like a matrix of more data nodes, again like a matrix of more data nodes, again like a matrix of numbers, high content image, something numbers, high content image, something numbers, high content image, something like this. A task prompt carefully like this. A task prompt carefully like this. A task prompt carefully describing some scientific goal, describing some scientific goal, describing some scientific goal, configuration for a grader, and then a configuration for a grader, and then a configuration for a grader, and then a deterministic grader, so a Python deterministic grader, so a Python deterministic grader, so a Python function. function. function. If you guys notice, this looks a lot If you guys notice, this looks a lot If you guys notice, this looks a lot like SweetBench. We borrowed a lot of like SweetBench. We borrowed a lot of like SweetBench. We borrowed a lot of the early ideas and tried to extend them the early ideas and tried to extend them the early ideas and tried to extend them as much as possible. Evaluation ends up as much as possible. Evaluation ends up as much as possible. Evaluation ends up looking like this. looking like this. looking like this. It's a lot of JSON. It's a lot of JSON. It's a lot of JSON. And we ended up identifying properties And we ended up identifying properties And we ended up identifying properties of like what we thought good biological of like what we thought good biological of like what we thought good biological tests were. tests were. tests were. Um Um Um little little different from code, and little little different from code, and little little different from code, and we built on these over time, but they we built on these over time, but they we built on these over time, but they still hold up. They got to be still hold up. They got to be still hold up. They got to be verifiable. You have to be able to check verifiable. You have to be able to check verifiable. You have to be able to check the success condition with a function. the success condition with a function. the success condition with a function. Um nothing's changed there. We'll get Um nothing's changed there. We'll get Um nothing's changed there. We'll get into some rubric stuff later, but still into some rubric stuff later, but still into some rubric stuff later, but still holds. Durability is particularly holds. Durability is particularly holds. Durability is particularly important. Science does not admit clear important. Science does not admit clear important. Science does not admit clear ground truth. Um if you are lazy with ground truth. Um if you are lazy with ground truth. Um if you are lazy with your ground truth construction of the your ground truth construction of the your ground truth construction of the task, a task, a task, a possible valid analysis path can come possible valid analysis path can come possible valid analysis path can come with the correct answer, um and you'll with the correct answer, um and you'll with the correct answer, um and you'll fail it uh incorrectly. So, you got to fail it uh incorrectly. So, you got to fail it uh incorrectly. So, you got to make sure you're reasoning about make sure you're reasoning about make sure you're reasoning about something that's somehow invariant something that's somehow invariant something that's somehow invariant across analysis paths.

across analysis paths. across analysis paths. And then obviously, we're we're working And then obviously, we're we're working And then obviously, we're we're working with the genetics stuff here. You don't with the genetics stuff here. You don't with the genetics stuff here. You don't want the model to answer the question in want the model to answer the question in want the model to answer the question in one turn. one turn. one turn. You You want the conclusion to require You You want the conclusion to require You You want the conclusion to require interaction with the data, not some interaction with the data, not some interaction with the data, not some memorized knowledge. In practice, that's memorized knowledge. In practice, that's memorized knowledge. In practice, that's pretty difficult. We learned a lot about pretty difficult. We learned a lot about pretty difficult. We learned a lot about what models could do and which ones to what models could do and which ones to what models could do and which ones to use in specific context for this use in specific context for this use in specific context for this category of work for our customers, and category of work for our customers, and category of work for our customers, and we thought, "Hey, this is pretty cool. we thought, "Hey, this is pretty cool. we thought, "Hey, this is pretty cool. Let's Let's start to improve and learn Let's Let's start to improve and learn Let's Let's start to improve and learn more about this benchmarking problem." more about this benchmarking problem." more about this benchmarking problem." So, we jumped to human verification long So, we jumped to human verification long So, we jumped to human verification long horizon extension. I'm going to quickly horizon extension. I'm going to quickly horizon extension. I'm going to quickly breeze through these. So, human breeze through these. So, human breeze through these. So, human verification is incredibly important in verification is incredibly important in verification is incredibly important in science. Science is not in the clear science. Science is not in the clear science. Science is not in the clear ground truth. ground truth. ground truth. Uh after watching trajectory data from Uh after watching trajectory data from Uh after watching trajectory data from multiple rounds of model releases, circa multiple rounds of model releases, circa multiple rounds of model releases, circa like January to March of this year, like January to March of this year, like January to March of this year, um we really realized a lot of our um we really realized a lot of our um we really realized a lot of our assumptions were pretty bad. Um and in assumptions were pretty bad. Um and in assumptions were pretty bad. Um and in the absence of like a canonical answer, the absence of like a canonical answer, the absence of like a canonical answer, uh having a bunch of scientists uh having a bunch of scientists uh having a bunch of scientists grade each other's work ended up being grade each other's work ended up being grade each other's work ended up being like the best proxy. like the best proxy. like the best proxy. So, So, So, I'm going to look at one one issue to I'm going to look at one one issue to I'm going to look at one one issue to highlight exactly what I'm talking highlight exactly what I'm talking highlight exactly what I'm talking about. Um this problem ambiguity. about. Um this problem ambiguity. about. Um this problem ambiguity. A task might ask an agent to split a A task might ask an agent to split a A task might ask an agent to split a gene list into two groups of activity, gene list into two groups of activity, gene list into two groups of activity, microglial activation, oligodendrocyte microglial activation, oligodendrocyte microglial activation, oligodendrocyte inflammation, just like biological inflammation, just like biological inflammation, just like biological categories of things. Score the cells, categories of things. Score the cells, categories of things. Score the cells, find neighboring oligodendrocytes around find neighboring oligodendrocytes around find neighboring oligodendrocytes around some region using an appropriate radius, some region using an appropriate radius, some region using an appropriate radius, compute a Spearman correlation at two compute a Spearman correlation at two compute a Spearman correlation at two time points.

time points. time points. Uh as you can probably clearly deduce, Uh as you can probably clearly deduce, Uh as you can probably clearly deduce, the original problem statement creates a the original problem statement creates a the original problem statement creates a host of open choices. How do you split host of open choices. How do you split host of open choices. How do you split the gene list? the gene list? the gene list? How do you count what inflammatory genes How do you count what inflammatory genes How do you count what inflammatory genes are? It's like somewhat ambiguous word. are? It's like somewhat ambiguous word. are? It's like somewhat ambiguous word. How do you normalize the data? Um what How do you normalize the data? Um what How do you normalize the data? Um what what what the hell is an appropriate what what the hell is an appropriate what what the hell is an appropriate radius? How do you pull the counts radius? How do you pull the counts radius? How do you pull the counts within the selected radius? within the selected radius? within the selected radius? Um Um Um These are all problems that These are all problems that These are all problems that pointed to tasks that were bad, that pointed to tasks that were bad, that pointed to tasks that were bad, that only became revealed with human only became revealed with human only became revealed with human verification. Another issue is just like verification. Another issue is just like verification. Another issue is just like a lot of people in bioinformatics a lot of people in bioinformatics a lot of people in bioinformatics canonically have used like numerical canonically have used like numerical canonically have used like numerical thresholds to QC stuff. Just like thresholds to QC stuff. Just like thresholds to QC stuff. Just like completely arbitrary stuff. Um cool completely arbitrary stuff. Um cool completely arbitrary stuff. Um cool thing about evaluation like coding is it thing about evaluation like coding is it thing about evaluation like coding is it forces you to reason about things more forces you to reason about things more forces you to reason about things more rigorously than you would when you're rigorously than you would when you're rigorously than you would when you're doing the thing yourself. If you have to doing the thing yourself. If you have to doing the thing yourself. If you have to teach a machine to do it, uh you you teach a machine to do it, uh you you teach a machine to do it, uh you you might be picking out some structure might be picking out some structure might be picking out some structure that's more important or more durable that's more important or more durable that's more important or more durable than what you were doing if if you're than what you were doing if if you're than what you were doing if if you're just doing it on your own. just doing it on your own. just doing it on your own. So, we just found a lot of these So, we just found a lot of these So, we just found a lot of these numerical thresholds to be like that. Uh numerical thresholds to be like that. Uh numerical thresholds to be like that. Uh I'm not going to get into this. I'm not going to get into this. I'm not going to get into this. Um after two rounds of human attempts, Um after two rounds of human attempts, Um after two rounds of human attempts, we produce a verified subset of the we produce a verified subset of the we produce a verified subset of the benchmark. benchmark. benchmark. Uh Uh Uh we we publish it. That was fun. Uh and we we publish it. That was fun. Uh and we we publish it. That was fun. Uh and then we also tried to increase the time then we also tried to increase the time then we also tried to increase the time horizon. So, I want to be clear the horizon. So, I want to be clear the horizon. So, I want to be clear the frontier of knowledge is still not quite frontier of knowledge is still not quite frontier of knowledge is still not quite there with biology. Like the labs are there with biology. Like the labs are there with biology. Like the labs are starting to catch up with the post starting to catch up with the post starting to catch up with the post training, but we kind of want to stay training, but we kind of want to stay training, but we kind of want to stay ahead.

ahead. ahead. Um so, we built a benchmark that we Um so, we built a benchmark that we Um so, we built a benchmark that we thought would recapitulate like really thought would recapitulate like really thought would recapitulate like really difficult true work. Um so, we built a difficult true work. Um so, we built a difficult true work. Um so, we built a space much longer. space much longer. space much longer. Um real biological tasks are messy. They Um real biological tasks are messy. They Um real biological tasks are messy. They use lots of different experiment types. use lots of different experiment types. use lots of different experiment types. They are They are They are use the whole workflow, don't use little use the whole workflow, don't use little use the whole workflow, don't use little chunks. They are uh tasks every step is chunks. They are uh tasks every step is chunks. They are uh tasks every step is interpreted against the experimental interpreted against the experimental interpreted against the experimental design or contextualized with some prior design or contextualized with some prior design or contextualized with some prior literature literature literature and the original goal of what you're and the original goal of what you're and the original goal of what you're doing in the first place. doing in the first place. doing in the first place. Um so, we built a bank of these tasks Um so, we built a bank of these tasks Um so, we built a bank of these tasks that are really trying to simulate the that are really trying to simulate the that are really trying to simulate the result sections of entire papers or the result sections of entire papers or the result sections of entire papers or the kinds of decisions you make in practice kinds of decisions you make in practice kinds of decisions you make in practice in industry to make a go no go decision in industry to make a go no go decision in industry to make a go no go decision on a drug program. on a drug program. on a drug program. Um Um Um these got these tasks took like a week these got these tasks took like a week these got these tasks took like a week for a group of three people to make for a group of three people to make for a group of three people to make each. each. each. Taught us a bunch of stuff. Taught us a bunch of stuff. Taught us a bunch of stuff. Um example is can an agent reconstruct a Um example is can an agent reconstruct a Um example is can an agent reconstruct a like metastatic niche in a tumor? Um if like metastatic niche in a tumor? Um if like metastatic niche in a tumor? Um if you have like a tumor biopsy and a bunch you have like a tumor biopsy and a bunch you have like a tumor biopsy and a bunch of metastatic biopsies from like where of metastatic biopsies from like where of metastatic biopsies from like where it metastasized and spread across the it metastasized and spread across the it metastasized and spread across the body, can it use like both the genetics body, can it use like both the genetics body, can it use like both the genetics and mRNA of the metastatic lesions and and mRNA of the metastatic lesions and and mRNA of the metastatic lesions and the tumor to like find the part of the the tumor to like find the part of the the tumor to like find the part of the tumor that initially seeded the tumor that initially seeded the tumor that initially seeded the metastatic growth and let it spread?

metastatic growth and let it spread? metastatic growth and let it spread? From that you can figure out like, "Hey, From that you can figure out like, "Hey, From that you can figure out like, "Hey, what parts of the tumor are more like what parts of the tumor are more like what parts of the tumor are more like genetically fit? Which ones actually genetically fit? Which ones actually genetically fit? Which ones actually cause problems?" cause problems?" cause problems?" And construct targeted medicines to nip And construct targeted medicines to nip And construct targeted medicines to nip them in the bud. Like for example, this them in the bud. Like for example, this them in the bud. Like for example, this is one of the benchmark uh evals in the is one of the benchmark uh evals in the is one of the benchmark uh evals in the long horizon set. None of the models get long horizon set. None of the models get long horizon set. None of the models get this right. Um but they're getting this right. Um but they're getting this right. Um but they're getting there. As As imagine with these long there. As As imagine with these long there. As As imagine with these long horizon extensions, horizon extensions, horizon extensions, uh verifiable reward at the end are like uh verifiable reward at the end are like uh verifiable reward at the end are like somewhat uninformative. So, somewhat uninformative. So, somewhat uninformative. So, we we're starting to play with rubrics, we we're starting to play with rubrics, we we're starting to play with rubrics, uh constructing these choke points. If uh constructing these choke points. If uh constructing these choke points. If you can imagine like the set of analysis you can imagine like the set of analysis you can imagine like the set of analysis paths is inducing some sort of tree. paths is inducing some sort of tree. paths is inducing some sort of tree. Um there are Um there are Um there are nodes that are invariant with respect to nodes that are invariant with respect to nodes that are invariant with respect to yeah, different paths, and you can use yeah, different paths, and you can use yeah, different paths, and you can use these to build rubrics um these to build rubrics um these to build rubrics um using knowledge of how the task work. using knowledge of how the task work. using knowledge of how the task work. Uh we we we're playing with these. Uh we we we're playing with these. Uh we we we're playing with these. We noticed that um they're associated We noticed that um they're associated We noticed that um they're associated with the verifiable outcomes, with the verifiable outcomes, with the verifiable outcomes, uh which is exciting, but they're uh which is exciting, but they're uh which is exciting, but they're loosely correlated numerically, um loosely correlated numerically, um loosely correlated numerically, um making us making us making us not fully have confidence in them for not fully have confidence in them for not fully have confidence in them for things like RL or benchmarking. Um a lot things like RL or benchmarking. Um a lot things like RL or benchmarking. Um a lot of lot more work to do here still. of lot more work to do here still. of lot more work to do here still. Uh we we still strongly believe the Uh we we still strongly believe the Uh we we still strongly believe the verifiability structure is what's going verifiability structure is what's going verifiability structure is what's going to to to carry carry carry in- intelligence uh in- intelligence uh in- intelligence uh a bit a bit longer.

a bit a bit longer. a bit a bit longer. And so, these days uh excitingly, we've And so, these days uh excitingly, we've And so, these days uh excitingly, we've been expanding um from this initial been expanding um from this initial been expanding um from this initial spatial focus. spatial focus. spatial focus. Uh Uh Uh really cool to see the frontier labs and really cool to see the frontier labs and really cool to see the frontier labs and community adopt these benchmarks community adopt these benchmarks community adopt these benchmarks organically. Um organically. Um organically. Um we had this interesting position by we had this interesting position by we had this interesting position by like, you know, building and shipping like, you know, building and shipping like, you know, building and shipping products uh early and kind of playing products uh early and kind of playing products uh early and kind of playing with the coding agents. So, I think we with the coding agents. So, I think we with the coding agents. So, I think we just had uh just had uh just had uh early advantage, but early advantage, but early advantage, but the benchmarks are now in like the the benchmarks are now in like the the benchmarks are now in like the recent Anthropic model cards, and uh recent Anthropic model cards, and uh recent Anthropic model cards, and uh this is a picture from yesterday. Eric this is a picture from yesterday. Eric this is a picture from yesterday. Eric just showing the benchmarks just showing the benchmarks just showing the benchmarks um at their cloud science launch. They um at their cloud science launch. They um at their cloud science launch. They don't tell us this happens. They just don't tell us this happens. They just don't tell us this happens. They just like do it, um and you like read about like do it, um and you like read about like do it, um and you like read about it, and it's cool. it, and it's cool. it, and it's cool. Uh we published a bunch more papers Uh we published a bunch more papers Uh we published a bunch more papers um beyond spatial to other omics um beyond spatial to other omics um beyond spatial to other omics classes, so other experiment types, classes, so other experiment types, classes, so other experiment types, single-cell, epigenomics, so RNA, single-cell, epigenomics, so RNA, single-cell, epigenomics, so RNA, and then the bit above the the DNA, and then the bit above the the DNA, and then the bit above the the DNA, and then long horizon extensions of and then long horizon extensions of and then long horizon extensions of these things. these things. these things. Um and then we're starting to index and Um and then we're starting to index and Um and then we're starting to index and measure measure measure the very gnarly complex landscape that the very gnarly complex landscape that the very gnarly complex landscape that is drug discovery. We just put out our is drug discovery. We just put out our is drug discovery. We just put out our first benchmark on um preclinical first benchmark on um preclinical first benchmark on um preclinical pharmacology for small molecules, pharmacology for small molecules, pharmacology for small molecules, and then systematically biting off and then systematically biting off and then systematically biting off pieces of the program landscape from pieces of the program landscape from pieces of the program landscape from discovery to development to translation, discovery to development to translation, discovery to development to translation, strati- stratifying it by strati- stratifying it by strati- stratifying it by um therapeutic types and experiment um therapeutic types and experiment um therapeutic types and experiment types.

types. types. We just acquired a uh company building We just acquired a uh company building We just acquired a uh company building in biosecurity to form a biosecurity in biosecurity to form a biosecurity in biosecurity to form a biosecurity team. team. team. And then we just put out a collaboration And then we just put out a collaboration And then we just put out a collaboration with American Wetware and a surveillance with American Wetware and a surveillance with American Wetware and a surveillance company called Aquid. company called Aquid. company called Aquid. Um some new work that was first released Um some new work that was first released Um some new work that was first released this morning. I don't know if you guys this morning. I don't know if you guys this morning. I don't know if you guys have been hearing have been hearing have been hearing uh fuzzles kind of suck in biology right uh fuzzles kind of suck in biology right uh fuzzles kind of suck in biology right now. now. now. If you ask Fable basic questions about If you ask Fable basic questions about If you ask Fable basic questions about like mitochondria, it'll won't answer. like mitochondria, it'll won't answer. like mitochondria, it'll won't answer. It's kind of stupid. So, I mean this is It's kind of stupid. So, I mean this is It's kind of stupid. So, I mean this is just like an evaluation problem. There's just like an evaluation problem. There's just like an evaluation problem. There's a lot There's a lot more nuance to this, a lot There's a lot more nuance to this, a lot There's a lot more nuance to this, but I just use that cuz people tend to but I just use that cuz people tend to but I just use that cuz people tend to recognize it. Uh where we build routine recognize it. Uh where we build routine recognize it. Uh where we build routine tasks that simulate the kinds of things tasks that simulate the kinds of things tasks that simulate the kinds of things a scientist would ask for, and then more a scientist would ask for, and then more a scientist would ask for, and then more sinister red team tasks, which are sinister red team tasks, which are sinister red team tasks, which are supposed to look innocuous but have some supposed to look innocuous but have some supposed to look innocuous but have some structure that is bad, like, "Hey, I structure that is bad, like, "Hey, I structure that is bad, like, "Hey, I want to clone a gene into a bacteria, want to clone a gene into a bacteria, want to clone a gene into a bacteria, and I'm telling you it's GFP." It's like and I'm telling you it's GFP." It's like and I'm telling you it's GFP." It's like a glowing protein, but in reality it's a glowing protein, but in reality it's a glowing protein, but in reality it's like like like a toxin um a toxin um a toxin um or it could be used to bootstrap a or it could be used to bootstrap a or it could be used to bootstrap a virus. We found that the routine tasks virus. We found that the routine tasks virus. We found that the routine tasks like drastic get drastic used like drastic get drastic used like drastic get drastic used drastically more frequently than the drastically more frequently than the drastically more frequently than the red team tasks, which is uh not great.

red team tasks, which is uh not great. red team tasks, which is uh not great. Um we're aggregating a lot of these Um we're aggregating a lot of these Um we're aggregating a lot of these results, uh results, uh results, uh essential resource along with all the essential resource along with all the essential resource along with all the preprints and a lot of evals and preprints and a lot of evals and preprints and a lot of evals and trajectories for you guys to check out. trajectories for you guys to check out. trajectories for you guys to check out. Um and I actually did okay on time. Um and I actually did okay on time. Um and I actually did okay on time. Let's go. Let's go. Let's go. Uh and that's it. So, we we are kind of Uh and that's it. So, we we are kind of Uh and that's it. So, we we are kind of like I don't even I hate the word lab, like I don't even I hate the word lab, like I don't even I hate the word lab, but we're kind of like a research lab but we're kind of like a research lab but we're kind of like a research lab for for for bio, and we do research and deployment bio, and we do research and deployment bio, and we do research and deployment of these agents. So, we we still have of these agents. So, we we still have of these agents. So, we we still have like a lot of customers. Um like a lot of customers. Um like a lot of customers. Um we work with the kit manufacturers, and we work with the kit manufacturers, and we work with the kit manufacturers, and we use that to inform what kinds of we use that to inform what kinds of we use that to inform what kinds of things we make benchmarks for. We try to things we make benchmarks for. We try to things we make benchmarks for. We try to get the labs to compete get the labs to compete get the labs to compete um on the benchmarks cuz then it makes um on the benchmarks cuz then it makes um on the benchmarks cuz then it makes the models better at our products. And the models better at our products. And the models better at our products. And it's a It's been a pretty rewarding it's a It's been a pretty rewarding it's a It's been a pretty rewarding flywheel. A lot of growth, and we're flywheel. A lot of growth, and we're flywheel. A lot of growth, and we're hiring aggressively across engineering hiring aggressively across engineering hiring aggressively across engineering and science. So, if you're interested in and science. So, if you're interested in and science. So, if you're interested in this work, please find me afterwards. this work, please find me afterwards. this work, please find me afterwards. Thank you.