How Anthropic Builds: Lessons from Labs — Mike Krieger, Anthropic

>> Joining us on stage is the co-founder of >> Joining us on stage is the co-founder of Instagram [music] Instagram [music] Instagram [music] and a member of technical staff at and a member of technical staff at and a member of technical staff at Anthropic. Anthropic. Anthropic. Mike Krieger. >> How's everybody doing? I mean, good >> How's everybody doing? I mean, good morning. morning. morning. Nice. Nice. Nice. Um Um Um >> Mike, thank you for releasing Fable just >> Mike, thank you for releasing Fable just >> Mike, thank you for releasing Fable just in time for us. in time for us. in time for us. >> Exactly for the conference. We timed it. >> Exactly for the conference. We timed it. >> Exactly for the conference. We timed it. >> [laughter] >> [laughter] >> [laughter] >> Um we're we're so glad to have you. Uh >> Um we're we're so glad to have you. Uh >> Um we're we're so glad to have you. Uh you're uh one of the preeminent builders you're uh one of the preeminent builders you're uh one of the preeminent builders and you're a leading labs at Anthropic. and you're a leading labs at Anthropic. and you're a leading labs at Anthropic. Um how has your Um how has your Um how has your model usage changed as as you've, you model usage changed as as you've, you model usage changed as as you've, you know, seen models internally grow? know, seen models internally grow? know, seen models internally grow? >> Yeah, I mean, for me it's been like both >> Yeah, I mean, for me it's been like both >> Yeah, I mean, for me it's been like both the model shift and then my role shift. the model shift and then my role shift. the model shift and then my role shift. So, I for like the first 2 years I was So, I for like the first 2 years I was So, I for like the first 2 years I was at Anthropic, I was chief product at Anthropic, I was chief product at Anthropic, I was chief product officer. And then I kept seeing people officer. And then I kept seeing people officer. And then I kept seeing people build with the models and the FOMO just build with the models and the FOMO just build with the models and the FOMO just kept increasing because I was you know, kept increasing because I was you know, kept increasing because I was you know, use the models as much as possible. But use the models as much as possible. But use the models as much as possible. But for example, on product strategy I would for example, on product strategy I would for example, on product strategy I would write a strategy doc and then have write a strategy doc and then have write a strategy doc and then have Claude critique it and maybe you can use Claude critique it and maybe you can use Claude critique it and maybe you can use a workflow, but it's not quite the same a workflow, but it's not quite the same a workflow, but it's not quite the same as like building in that pure way. And I as like building in that pure way. And I as like building in that pure way. And I was like spending all my weekends trying was like spending all my weekends trying was like spending all my weekends trying to build with it and I realized, "Okay, to build with it and I realized, "Okay, to build with it and I realized, "Okay, I actually just need to shift. It's like I actually just need to shift. It's like I actually just need to shift. It's like way too interesting a time." And it's way too interesting a time." And it's way too interesting a time." And it's actually an interesting trend I've seen actually an interesting trend I've seen actually an interesting trend I've seen now like several people that were CTOs now like several people that were CTOs now like several people that were CTOs at other places are like now joining as at other places are like now joining as at other places are like now joining as ICs at Anthropic and other places. But I ICs at Anthropic and other places. But I ICs at Anthropic and other places. But I made a role shift and it was actually made a role shift and it was actually made a role shift and it was actually right around the time where we started right around the time where we started right around the time where we started getting sort of internal snapshots of

getting sort of internal snapshots of getting sort of internal snapshots of what became Mythos and Fable. And what what became Mythos and Fable. And what what became Mythos and Fable. And what was really interesting watching that was really interesting watching that was really interesting watching that sort of shift was um that sort of shift was um that sort of shift was um that kind of change between I have an idea, kind of change between I have an idea, kind of change between I have an idea, I'm going to like sort of break it down I'm going to like sort of break it down I'm going to like sort of break it down in my head much more how I would do in my head much more how I would do in my head much more how I would do engineering normally, and then kind of engineering normally, and then kind of engineering normally, and then kind of iterate through these different steps to iterate through these different steps to iterate through these different steps to moving to much more of the paradigm of moving to much more of the paradigm of moving to much more of the paradigm of I'm going to describe the goal, like go I'm going to describe the goal, like go I'm going to describe the goal, like go off and work on it, and then like we can off and work on it, and then like we can off and work on it, and then like we can talk about what trade-offs you you know, talk about what trade-offs you you know, talk about what trade-offs you you know, surface some questions along the way, surface some questions along the way, surface some questions along the way, but then but then but then figure out what where you landed and figure out what where you landed and figure out what where you landed and where we can go from there. I find it's where we can go from there. I find it's where we can go from there. I find it's hard. I don't know if people have this hard. I don't know if people have this hard. I don't know if people have this experience where experience where experience where people people people people's only been re-enabled for a people's only been re-enabled for a people's only been re-enabled for a couple of days. People's definitely way couple of days. People's definitely way couple of days. People's definitely way way smarter than me. So, sometimes it'll way smarter than me. So, sometimes it'll way smarter than me. So, sometimes it'll finish work and be like, here's the finish work and be like, here's the finish work and be like, here's the trade-offs I made. I'm like, can you trade-offs I made. I'm like, can you trade-offs I made. I'm like, can you explain it to me like I'm a little explain it to me like I'm a little explain it to me like I'm a little dumber than you are because I need you dumber than you are because I need you dumber than you are because I need you to like sort of break this down for me. to like sort of break this down for me. to like sort of break this down for me. But, that's been one sort of big change But, that's been one sort of big change But, that's been one sort of big change is sort of moving from that task is sort of moving from that task is sort of moving from that task delegation to like express the end state delegation to like express the end state delegation to like express the end state and then have it go and and cook on it. and then have it go and and cook on it. and then have it go and and cook on it. >> Yeah, >> Yeah, >> Yeah, we're all learning how to delegate we're all learning how to delegate we're all learning how to delegate better. Tariq did us a huge favor better. Tariq did us a huge favor better. Tariq did us a huge favor yesterday. yesterday. yesterday. We We We Did you want to read it in the Did you want to read it in the Did you want to read it in the newspaper?

newspaper? newspaper? You know, that we have we have You know, that we have we have You know, that we have we have write-ups of talks now in in like the write-ups of talks now in in like the write-ups of talks now in in like the next day's newspaper. next day's newspaper. next day's newspaper. >> He said, be unreasonable. >> He said, be unreasonable. >> He said, be unreasonable. In what ways you know, you know, have In what ways you know, you know, have In what ways you know, you know, have you been more ambitious? you been more ambitious? you been more ambitious? >> Yeah, you're prompting. >> Yeah, you're prompting. >> Yeah, you're prompting. >> I love that I I mean, I love that >> I love that I I mean, I love that >> I love that I I mean, I love that framing. We actually just hit this framing. We actually just hit this framing. We actually just hit this today. I'm one of the labs initiatives I today. I'm one of the labs initiatives I today. I'm one of the labs initiatives I have is internal product, and somebody have is internal product, and somebody have is internal product, and somebody was like, hey, it doesn't work the way I was like, hey, it doesn't work the way I was like, hey, it doesn't work the way I want it to, want it to, want it to, and can you make some changes? And I and can you make some changes? And I and can you make some changes? And I realized, oh, I'm just going to go ask realized, oh, I'm just going to go ask realized, oh, I'm just going to go ask Claude to do this. Like, why don't you Claude to do this. Like, why don't you Claude to do this. Like, why don't you ask Claude? And this was a non-technical ask Claude? And this was a non-technical ask Claude? And this was a non-technical person. So, I actually think as an person. So, I actually think as an person. So, I actually think as an industry or even as a product team, we industry or even as a product team, we industry or even as a product team, we have to teach people to be more have to teach people to be more have to teach people to be more unreasonable in their usage, and it's unreasonable in their usage, and it's unreasonable in their usage, and it's sort of hard to imagine. I think that sort of hard to imagine. I think that sort of hard to imagine. I think that that that that if I can digress for a second on product if I can digress for a second on product if I can digress for a second on product design, I think right now the like kind design, I think right now the like kind design, I think right now the like kind of first generation of AI products, we of first generation of AI products, we of first generation of AI products, we put them too much in a box and constrain put them too much in a box and constrain put them too much in a box and constrain their their sort of access to tools or their their sort of access to tools or their their sort of access to tools or kind of degrees of freedom, which means kind of degrees of freedom, which means kind of degrees of freedom, which means it was much harder to be unreasonable, it was much harder to be unreasonable, it was much harder to be unreasonable, right? When you say, do this thing for right? When you say, do this thing for right? When you say, do this thing for me, and then it would be like, well, I me, and then it would be like, well, I me, and then it would be like, well, I can't. I can barely like I can write can't. I can barely like I can write can't. I can barely like I can write code, but I can't really run it, or I code, but I can't really run it, or I code, but I can't really run it, or I can kind of introspect my environment, can kind of introspect my environment, can kind of introspect my environment, but not really.

but not really. but not really. Um and I think as you see our own like Um and I think as you see our own like Um and I think as you see our own like product progression even with things product progression even with things product progression even with things like co-work where like, you know, does like co-work where like, you know, does like co-work where like, you know, does every single like knowledge worker need every single like knowledge worker need every single like knowledge worker need a virtual machine that can write bash? a virtual machine that can write bash? a virtual machine that can write bash? Like, on the face of it, no, but then Like, on the face of it, no, but then Like, on the face of it, no, but then when you realize, oh, actually, that way when you realize, oh, actually, that way when you realize, oh, actually, that way it can remediate an issue where, oh, I it can remediate an issue where, oh, I it can remediate an issue where, oh, I tried to parse a PDF using our built-in tried to parse a PDF using our built-in tried to parse a PDF using our built-in PDF parser. I hit this yesterday and it PDF parser. I hit this yesterday and it PDF parser. I hit this yesterday and it was like, ah, I can't parse it this way. was like, ah, I can't parse it this way. was like, ah, I can't parse it this way. Well, okay, well, I can probably write a Well, okay, well, I can probably write a Well, okay, well, I can probably write a script that can do this as well. Um so, script that can do this as well. Um so, script that can do this as well. Um so, I think that's it. My most unreasonable I think that's it. My most unreasonable I think that's it. My most unreasonable thing though was uh one of our labs thing though was uh one of our labs thing though was uh one of our labs projects I wrote in Python like near and projects I wrote in Python like near and projects I wrote in Python like near and dear to my heart. All of Instagram was dear to my heart. All of Instagram was dear to my heart. All of Instagram was in Python. But I think they're finally in Python. But I think they're finally in Python. But I think they're finally converting it to PHP now that they have converting it to PHP now that they have converting it to PHP now that they have um like models that can do it. I know. um like models that can do it. I know. um like models that can do it. I know. >> [laughter] >> [laughter] >> [laughter] >> Tokens. Um and uh for deployment I >> Tokens. Um and uh for deployment I >> Tokens. Um and uh for deployment I realized that Cloud Code had like realized that Cloud Code had like realized that Cloud Code had like figured out a better deployment story figured out a better deployment story figured out a better deployment story with Bun. And I was like, okay, I need with Bun. And I was like, okay, I need with Bun. And I was like, okay, I need to port this whole thing from Python to to port this whole thing from Python to to port this whole thing from Python to TypeScript. Like, as a, you know, if I TypeScript. Like, as a, you know, if I TypeScript. Like, as a, you know, if I put on my like 2010s engineering hat or put on my like 2010s engineering hat or put on my like 2010s engineering hat or even my early 20 20s, like, that's a even my early 20 20s, like, that's a even my early 20 20s, like, that's a dumb idea. Like, who would ever port dumb idea. Like, who would ever port dumb idea. Like, who would ever port like, at that point, you know, a couple like, at that point, you know, a couple like, at that point, you know, a couple hundred thousands of lines of code. Um hundred thousands of lines of code. Um hundred thousands of lines of code. Um but I was like, I think this is doable but I was like, I think this is doable but I was like, I think this is doable now and I basically created this dynamic now and I basically created this dynamic now and I basically created this dynamic workflow setup and over the weekend had workflow setup and over the weekend had workflow setup and over the weekend had it port the whole thing, like, verify it port the whole thing, like, verify it port the whole thing, like, verify it, double-check it, then read both code it, double-check it, then read both code it, double-check it, then read both code like this basically churn and churn and like this basically churn and churn and like this basically churn and churn and churn and then came back Monday to a churn and then came back Monday to a churn and then came back Monday to a completed workflow that was a ported completed workflow that was a ported completed workflow that was a ported version of that thing. So, that probably version of that thing. So, that probably version of that thing. So, that probably ranks on like the more unreasonable ranks on like the more unreasonable ranks on like the more unreasonable things. Like, yeah, just port this things. Like, yeah, just port this things. Like, yeah, just port this entire Python code base to TypeScript, entire Python code base to TypeScript, entire Python code base to TypeScript, get it working, get it deployable in, get it working, get it deployable in, get it working, get it deployable in, you know, a weekend.

you know, a weekend. you know, a weekend. >> Yeah, I mean, a lot of people are >> Yeah, I mean, a lot of people are >> Yeah, I mean, a lot of people are talking about the the Bun Zig to Rust talking about the the Bun Zig to Rust talking about the the Bun Zig to Rust version. I think a lot of people are version. I think a lot of people are version. I think a lot of people are also like, well, it's a compiler, it's a also like, well, it's a compiler, it's a also like, well, it's a compiler, it's a it's a runtime, it's got lots of tests, it's a runtime, it's got lots of tests, it's a runtime, it's got lots of tests, easy to do. Can you port Instagram, easy to do. Can you port Instagram, easy to do. Can you port Instagram, which you would know very well, to PHP which you would know very well, to PHP which you would know very well, to PHP like that? Like a like a product. like that? Like a like a product. like that? Like a like a product. >> Yeah, I mean, I think the product side >> Yeah, I mean, I think the product side >> Yeah, I mean, I think the product side of it it's even I don't know if it's of it it's even I don't know if it's of it it's even I don't know if it's easier or harder. One of the things we easier or harder. One of the things we easier or harder. One of the things we did at Instagram, this is when Python 3 did at Instagram, this is when Python 3 did at Instagram, this is when Python 3 came out and we were able to add type came out and we were able to add type came out and we were able to add type hints for the first time and it was hints for the first time and it was hints for the first time and it was people had a lot of internal people had a lot of internal people had a lot of internal conversations like, are we going to run conversations like, are we going to run conversations like, are we going to run out of steam on Python? And my out of steam on Python? And my out of steam on Python? And my perspective was always like, I think we perspective was always like, I think we perspective was always like, I think we can take this way further than we think can take this way further than we think can take this way further than we think we can, uh, but I think types are going we can, uh, but I think types are going we can, uh, but I think types are going to help us not sort of be in our own to help us not sort of be in our own to help us not sort of be in our own way. And we built this thing called way. And we built this thing called way. And we built this thing called Monkey Type where we basically like Monkey Type where we basically like Monkey Type where we basically like captured runtime type like basically the captured runtime type like basically the captured runtime type like basically the types that were actually getting used in types that were actually getting used in types that were actually getting used in production and then map those back to to production and then map those back to to production and then map those back to to the types in the code base. And I think the types in the code base. And I think the types in the code base. And I think because of that sort of pattern, I think because of that sort of pattern, I think because of that sort of pattern, I think there's really interesting ways in which there's really interesting ways in which there's really interesting ways in which if you're doing sort of conversion or if you're doing sort of conversion or if you're doing sort of conversion or sort of cross compiling using LLMs, you sort of cross compiling using LLMs, you sort of cross compiling using LLMs, you can also lean on production data a lot can also lean on production data a lot can also lean on production data a lot more or run sort of like segmented more or run sort of like segmented more or run sort of like segmented tests. I think that like there's a lot tests. I think that like there's a lot tests. I think that like there's a lot of, uh, things you can do there. But of, uh, things you can do there. But of, uh, things you can do there. But yeah, I think it's, I mean, the sky's yeah, I think it's, I mean, the sky's yeah, I think it's, I mean, the sky's the limit there as well. I think the the limit there as well. I think the the limit there as well. I think the hardest part is always finding the hardest part is always finding the hardest part is always finding the boundary around where you can start boundary around where you can start boundary around where you can start doing it incrementally without trying to doing it incrementally without trying to doing it incrementally without trying to boil the whole ocean and like swap it boil the whole ocean and like swap it boil the whole ocean and like swap it overnight.

overnight. overnight. >> Yeah, I mean, your users are your test >> Yeah, I mean, your users are your test >> Yeah, I mean, your users are your test ultimately and, um, ultimately and, um, ultimately and, um, you know, I we I also read another you know, I we I also read another you know, I we I also read another article in the newspaper about how you article in the newspaper about how you article in the newspaper about how you could just use rollouts and sometimes could just use rollouts and sometimes could just use rollouts and sometimes you don't really know, uh, what you're you don't really know, uh, what you're you don't really know, uh, what you're going to need it for, but when that going to need it for, but when that going to need it for, but when that infrastructure exists for your infrastructure exists for your infrastructure exists for your experiments and to roll things out, it's experiments and to roll things out, it's experiments and to roll things out, it's enabled so much. enabled so much. enabled so much. >> Yeah, I mean, I always found this was >> Yeah, I mean, I always found this was >> Yeah, I mean, I always found this was advice we got. It's like we launched advice we got. It's like we launched advice we got. It's like we launched Instagram and the happened to be the Instagram and the happened to be the Instagram and the happened to be the first week everything melted cuz we first week everything melted cuz we first week everything melted cuz we didn't really know what we were doing on didn't really know what we were doing on didn't really know what we were doing on the back end side of things. And, uh, the back end side of things. And, uh, the back end side of things. And, uh, coincidentally that week there was like coincidentally that week there was like coincidentally that week there was like a lunch that one of our investors just a lunch that one of our investors just a lunch that one of our investors just scheduled like not even for us. It was scheduled like not even for us. It was scheduled like not even for us. It was just a just a just a infrastructure lunch. And we ended up infrastructure lunch. And we ended up infrastructure lunch. And we ended up spending we totally like monopolized spending we totally like monopolized spending we totally like monopolized that conversation cuz everybody had that conversation cuz everybody had that conversation cuz everybody had their own opinion about how we could fix their own opinion about how we could fix their own opinion about how we could fix our scaling. Um, and like the two pieces our scaling. Um, and like the two pieces our scaling. Um, and like the two pieces of advice I got there is like 2010 that of advice I got there is like 2010 that of advice I got there is like 2010 that I like will forever retain is like, um, I like will forever retain is like, um, I like will forever retain is like, um, like basically like like basically like like basically like pre-measure everything that you think pre-measure everything that you think pre-measure everything that you think you might even remotely need because the you might even remotely need because the you might even remotely need because the worst thing is an outage where you're worst thing is an outage where you're worst thing is an outage where you're like, well, is this like number normal like, well, is this like number normal like, well, is this like number normal or is it high? And like, oh, I don't or is it high? And like, oh, I don't or is it high? And like, oh, I don't know because I don't have data until I know because I don't have data until I know because I don't have data until I just added this metric. And the other just added this metric. And the other just added this metric. And the other one is being like really thoughtful one is being like really thoughtful one is being like really thoughtful about knobs and feature flags. So even, about knobs and feature flags. So even, about knobs and feature flags. So even, you know, early Instagram we had like a you know, early Instagram we had like a you know, early Instagram we had like a very, uh, simple but really effective very, uh, simple but really effective very, uh, simple but really effective like way in which you could do like ramp like way in which you could do like ramp like way in which you could do like ramp outs and rollouts. And dynamic config outs and rollouts. And dynamic config outs and rollouts. And dynamic config too where, you know, a lot of our too where, you know, a lot of our too where, you know, a lot of our runtime configurations had to be runtime configurations had to be runtime configurations had to be changed, you know, in a matter of changed, you know, in a matter of changed, you know, in a matter of seconds so that we could handle load and seconds so that we could handle load and seconds so that we could handle load and being able to like do that in a first being able to like do that in a first being able to like do that in a first class way was was really important. I'm class way was was really important. I'm class way was was really important. I'm seeing that definitely in in AI as well seeing that definitely in in AI as well seeing that definitely in in AI as well where, you know, we're making all sorts where, you know, we're making all sorts where, you know, we're making all sorts of different trade-offs and having that of different trade-offs and having that of different trade-offs and having that kind of runtime configuration is super kind of runtime configuration is super kind of runtime configuration is super key.

key. key. >> Yeah. >> Yeah. >> Yeah. Uh my my favorite scaling story of Uh my my favorite scaling story of Uh my my favorite scaling story of Instagram by the way, I think it's like Instagram by the way, I think it's like Instagram by the way, I think it's like your launch day when you you DDoS your launch day when you you DDoS your launch day when you you DDoS yourself with the email. yourself with the email. yourself with the email. >> Yes. >> Yes. >> Yes. >> Which I people should look up that story >> Which I people should look up that story >> Which I people should look up that story if uh if you haven't seen it. Um I if uh if you haven't seen it. Um I if uh if you haven't seen it. Um I wanted to go into tags. Uh wanted to go into tags. Uh wanted to go into tags. Uh very very major shift. Uh it's it's how very very major shift. Uh it's it's how very very major shift. Uh it's it's how 60 something percent of your code is 60 something percent of your code is 60 something percent of your code is written today. written today. written today. >> Yeah. >> Yeah. >> Yeah. >> Um how do you square that with >> Um how do you square that with >> Um how do you square that with everything you just said where it's like everything you just said where it's like everything you just said where it's like very dynamic? Like you don't actually very dynamic? Like you don't actually very dynamic? Like you don't actually ship one app, you ship one app with ship one app, you ship one app with ship one app, you ship one app with 3,000 flags. 3,000 flags. 3,000 flags. >> Yeah. >> Yeah. >> Yeah. >> And like, well, what are you working on >> And like, well, what are you working on >> And like, well, what are you working on today? I don't know. Like it's it's for today? I don't know. Like it's it's for today? I don't know. Like it's it's for this segment of the population. this segment of the population. this segment of the population. >> Yeah. Yeah, I mean, I think there's a >> Yeah. Yeah, I mean, I think there's a >> Yeah. Yeah, I mean, I think there's a bunch of things. So, like with I was bunch of things. So, like with I was bunch of things. So, like with I was really excited. I was talking to Swix really excited. I was talking to Swix really excited. I was talking to Swix earlier like, I'm really excited that we earlier like, I'm really excited that we earlier like, I'm really excited that we have tag out there because it is uh how have tag out there because it is uh how have tag out there because it is uh how we've been working for a while and I we've been working for a while and I we've been working for a while and I would get up on stages and people like, would get up on stages and people like, would get up on stages and people like, "How do you work at Anthropic?" And I'd "How do you work at Anthropic?" And I'd "How do you work at Anthropic?" And I'd be like, "Oh, yeah, we use these things be like, "Oh, yeah, we use these things be like, "Oh, yeah, we use these things like that are not quite Claude code, but like that are not quite Claude code, but like that are not quite Claude code, but you know, uh but it's hard to describe you know, uh but it's hard to describe you know, uh but it's hard to describe it, but I mean, if you like got to poke it, but I mean, if you like got to poke it, but I mean, if you like got to poke into Anthropic, like you would see uh of into Anthropic, like you would see uh of into Anthropic, like you would see uh of course Claude code usage for things that course Claude code usage for things that course Claude code usage for things that are like more interactive or if you're are like more interactive or if you're are like more interactive or if you're kind of iterating on a particular uh kind of iterating on a particular uh kind of iterating on a particular uh sort of sort of sort of sort of specific thing where you want a sort of specific thing where you want a sort of specific thing where you want a lot of like a high sort of bandwidth lot of like a high sort of bandwidth lot of like a high sort of bandwidth back and forth, but most usage is back and forth, but most usage is back and forth, but most usage is actually much more delegating uh via actually much more delegating uh via actually much more delegating uh via tagging and via tag. And you can say tagging and via tag. And you can say tagging and via tag. And you can say like, "Here's the And the reason it's like, "Here's the And the reason it's like, "Here's the And the reason it's really interesting is how multiplayer it really interesting is how multiplayer it really interesting is how multiplayer it is." And it reminds me sort of of like is." And it reminds me sort of of like is." And it reminds me sort of of like um actually like Midjourney, like the um actually like Midjourney, like the um actually like Midjourney, like the fact that everyone was on Discord seeing fact that everyone was on Discord seeing fact that everyone was on Discord seeing how other people were using it. I think how other people were using it. I think how other people were using it. I think it actually to your earlier question it actually to your earlier question it actually to your earlier question really helps with that unreasonableness really helps with that unreasonableness really helps with that unreasonableness or ambition where the first time you see or ambition where the first time you see or ambition where the first time you see somebody tag Claude and be like, "Hey, somebody tag Claude and be like, "Hey, somebody tag Claude and be like, "Hey, you know, don't just fix this bug, but you know, don't just fix this bug, but you know, don't just fix this bug, but like now you are responsible for this

like now you are responsible for this like now you are responsible for this part of the code base and I want you to part of the code base and I want you to part of the code base and I want you to monitor this feedback channel and monitor this feedback channel and monitor this feedback channel and proactively take on tasks and then fix proactively take on tasks and then fix proactively take on tasks and then fix them and then also take like, you know, them and then also take like, you know, them and then also take like, you know, if this API changes, do that." Like I if this API changes, do that." Like I if this API changes, do that." Like I saw somebody do that. I was like, "Oh, saw somebody do that. I was like, "Oh, saw somebody do that. I was like, "Oh, wait, I've I've totally underutilizing wait, I've I've totally underutilizing wait, I've I've totally underutilizing this thing. I've just been using it as this thing. I've just been using it as this thing. I've just been using it as like a glorified Claude code and slack. like a glorified Claude code and slack. like a glorified Claude code and slack. Like that's definitely a totally like Like that's definitely a totally like Like that's definitely a totally like sort of new version of it, right? And sort of new version of it, right? And sort of new version of it, right? And then more advanced version is really then more advanced version is really then more advanced version is really trying to start thinking of it as a trying to start thinking of it as a trying to start thinking of it as a teammate that is actually sort of holds teammate that is actually sort of holds teammate that is actually sort of holds context, has memory, and can be context, has memory, and can be context, has memory, and can be proactive. And that's just really proactive. And that's just really proactive. And that's just really changed how we operate internally. It's changed how we operate internally. It's changed how we operate internally. It's much more like this multiplayer async much more like this multiplayer async much more like this multiplayer async proactive way than it is a you know, proactive way than it is a you know, proactive way than it is a you know, most people often their own CLIs. most people often their own CLIs. most people often their own CLIs. >> Are you bottlenecked by code review and >> Are you bottlenecked by code review and >> Are you bottlenecked by code review and get? Obviously, there is code review, get? Obviously, there is code review, get? Obviously, there is code review, but someone usually still looks at it. but someone usually still looks at it. but someone usually still looks at it. Is there a world in which you just merge Is there a world in which you just merge Is there a world in which you just merge it in? it in? it in? >> Yeah, we're it's a really good question. >> Yeah, we're it's a really good question. >> Yeah, we're it's a really good question. We are definitely still bottlenecked on We are definitely still bottlenecked on We are definitely still bottlenecked on reviews, especially for things that are reviews, especially for things that are reviews, especially for things that are like touching some architecture pieces. like touching some architecture pieces. like touching some architecture pieces. And it's actually more subtle than just And it's actually more subtle than just And it's actually more subtle than just being bottlenecked on review, cuz that's being bottlenecked on review, cuz that's being bottlenecked on review, cuz that's you know, okay, we can carve out time you know, okay, we can carve out time you know, okay, we can carve out time differently. It's like bottlenecked on differently. It's like bottlenecked on differently. It's like bottlenecked on human ability to even human ability to even human ability to even like fully conceptualize what we're like fully conceptualize what we're like fully conceptualize what we're doing. So, one of the reasons we built doing. So, one of the reasons we built doing. So, one of the reasons we built Claude code artifacts that we shipped a Claude code artifacts that we shipped a Claude code artifacts that we shipped a couple weeks ago was partially for that, couple weeks ago was partially for that, couple weeks ago was partially for that, which is which is which is you would send somebody a PR, and then you would send somebody a PR, and then you would send somebody a PR, and then they'd be like, I don't know, man. This they'd be like, I don't know, man. This they'd be like, I don't know, man. This is like 2,000 lines of code. Like, it is like 2,000 lines of code. Like, it is like 2,000 lines of code. Like, it looks like code to me. And what we looks like code to me. And what we looks like code to me. And what we started doing instead is sharing much started doing instead is sharing much started doing instead is sharing much more like, here's a Claude code more like, here's a Claude code more like, here's a Claude code artifact. Like, here's the explanation.

artifact. Like, here's the explanation. artifact. Like, here's the explanation. Here's the intention of the the change. Here's the intention of the the change. Here's the intention of the the change. Here's the trade-offs that were made. Here's the trade-offs that were made. Here's the trade-offs that were made. And like, I think that's going to be And like, I think that's going to be And like, I think that's going to be much more be the trend by which we much more be the trend by which we much more be the trend by which we communicate, which is the code is communicate, which is the code is communicate, which is the code is ultimately, you know, verifiable using ultimately, you know, verifiable using ultimately, you know, verifiable using some things, but actually like some things, but actually like some things, but actually like discussing intent and trade-offs, and discussing intent and trade-offs, and discussing intent and trade-offs, and then measuring in production I think then measuring in production I think then measuring in production I think that at least the direction of travel that at least the direction of travel that at least the direction of travel we've we've gone. And I don't review we've we've gone. And I don't review we've we've gone. And I don't review when I get a pull request, I wish I when I get a pull request, I wish I when I get a pull request, I wish I could say I reviewed every line of code. could say I reviewed every line of code. could say I reviewed every line of code. I definitely do not. I like actually I definitely do not. I like actually I definitely do not. I like actually talk to Claude about the the code and talk to Claude about the the code and talk to Claude about the the code and say, all right, like, these are the say, all right, like, these are the say, all right, like, these are the questions that I would have. Can you go questions that I would have. Can you go questions that I would have. Can you go investigate it? So, it is kind of investigate it? So, it is kind of investigate it? So, it is kind of Claude-powered code review, but still Claude-powered code review, but still Claude-powered code review, but still human-driven. And and for the really human-driven. And and for the really human-driven. And and for the really important ones. And for the ones that important ones. And for the ones that important ones. And for the ones that are like cosmetic visual changes, it's are like cosmetic visual changes, it's are like cosmetic visual changes, it's much more like look like we'll fix much more like look like we'll fix much more like look like we'll fix forward if we need to fix forward, you forward if we need to fix forward, you forward if we need to fix forward, you know. know. know. >> Yeah, totally. I think a lot of people >> Yeah, totally. I think a lot of people >> Yeah, totally. I think a lot of people are here are trying to figure that out, are here are trying to figure that out, are here are trying to figure that out, too. too. too. I wanted to talk also a little bit about I wanted to talk also a little bit about I wanted to talk also a little bit about Anthropic Labs in general. Anthropic Labs in general. Anthropic Labs in general. Nilay Patel, Nilay Patel, Nilay Patel, who you've probably met before, loves to who you've probably met before, loves to who you've probably met before, loves to ask ask the question like draw the org ask ask the question like draw the org ask ask the question like draw the org chart. chart. chart. Like how like people, you know, you ship Like how like people, you know, you ship Like how like people, you know, you ship your org chart. Like I think it's your org chart. Like I think it's your org chart. Like I think it's important like everyone knows cloud important like everyone knows cloud important like everyone knows cloud code. Now you've got tags.

code. Now you've got tags. code. Now you've got tags. Um How are you structuring the labs? Um How are you structuring the labs? Um How are you structuring the labs? >> Yeah, it's a good question. Because what >> Yeah, it's a good question. Because what >> Yeah, it's a good question. Because what we were trying to wrestle with was you we were trying to wrestle with was you we were trying to wrestle with was you want sort of people to be supported want sort of people to be supported want sort of people to be supported like, you know, I think the death of the like, you know, I think the death of the like, you know, I think the death of the engineering manager discipline has been engineering manager discipline has been engineering manager discipline has been greatly exaggerated. Like I think greatly exaggerated. Like I think greatly exaggerated. Like I think there's still a lot of coaching and there's still a lot of coaching and there's still a lot of coaching and interpersonal pieces and personal interpersonal pieces and personal interpersonal pieces and personal development that I think is still development that I think is still development that I think is still really, really important. But especially really, really important. But especially really, really important. But especially in a labs type group where like our in a labs type group where like our in a labs type group where like our whole cadence is two-week reviews where whole cadence is two-week reviews where whole cadence is two-week reviews where every project goes up for we call it every project goes up for we call it every project goes up for we call it persevere or pivot. So basically every persevere or pivot. So basically every persevere or pivot. So basically every project is up for review and either it's project is up for review and either it's project is up for review and either it's time to, you know, keep going, time to, you know, keep going, time to, you know, keep going, persevering, or you know, it's time to persevering, or you know, it's time to persevering, or you know, it's time to pivot it or even shut down. And, you pivot it or even shut down. And, you pivot it or even shut down. And, you know, we've shut down projects basically know, we've shut down projects basically know, we've shut down projects basically every single one of those cycles and every single one of those cycles and every single one of those cycles and it's like the more you do it, the less it's like the more you do it, the less it's like the more you do it, the less it's just like, "Oh no, my project is it's just like, "Oh no, my project is it's just like, "Oh no, my project is shut down. I failed." It's like, "No, shut down. I failed." It's like, "No, shut down. I failed." It's like, "No, that is definitely the intention of the that is definitely the intention of the that is definitely the intention of the labs team is to prototype quickly, try labs team is to prototype quickly, try labs team is to prototype quickly, try to ship internally, maybe get it to to ship internally, maybe get it to to ship internally, maybe get it to early access, and if it doesn't work, early access, and if it doesn't work, early access, and if it doesn't work, wind it down." But because of that kind wind it down." But because of that kind wind it down." But because of that kind of like rapid iteration, it means that of like rapid iteration, it means that of like rapid iteration, it means that if you align the org chart too much to if you align the org chart too much to if you align the org chart too much to the individual projects, you're going to the individual projects, you're going to the individual projects, you're going to end up like re-orging every two weeks, end up like re-orging every two weeks, end up like re-orging every two weeks, which would be a total nightmare. And so which would be a total nightmare. And so which would be a total nightmare. And so we've actually ended up with this we've actually ended up with this we've actually ended up with this interesting setup where like the the pod interesting setup where like the the pod interesting setup where like the the pod or the team that is working on a given or the team that is working on a given or the team that is working on a given we call them bets within labs, we call them bets within labs, we call them bets within labs, definitely just draws upon like all definitely just draws upon like all definitely just draws upon like all right, somebody from product, somebody right, somebody from product, somebody right, somebody from product, somebody from the eng team, from the eng team, from the eng team, you know, I'll jump in when it's a you know, I'll jump in when it's a you know, I'll jump in when it's a product I'm particularly interested in.

product I'm particularly interested in. product I'm particularly interested in. I'll come in and work together with the I'll come in and work together with the I'll come in and work together with the team on it. And that's the unit for that team on it. And that's the unit for that team on it. And that's the unit for that time. And there is the concept of a bet time. And there is the concept of a bet time. And there is the concept of a bet lead or a directly responsible lead or a directly responsible lead or a directly responsible individual. But the interesting thing is individual. But the interesting thing is individual. But the interesting thing is that they don't manage usually any of that they don't manage usually any of that they don't manage usually any of the other people, which kind of breaks the other people, which kind of breaks the other people, which kind of breaks the that kind of previous way in which a the that kind of previous way in which a the that kind of previous way in which a lot of these things were done. But I lot of these things were done. But I lot of these things were done. But I think it leaves it leaves us to be think it leaves it leaves us to be think it leaves it leaves us to be really flexible when you say, "Okay, really flexible when you say, "Okay, really flexible when you say, "Okay, actually this project is not going to actually this project is not going to actually this project is not going to work out. Let's disband and keep going work out. Let's disband and keep going work out. Let's disband and keep going and it's not a big deal." And the engine and it's not a big deal." And the engine and it's not a big deal." And the engine manager is much more playing the like manager is much more playing the like manager is much more playing the like make sure every individual is assigned make sure every individual is assigned make sure every individual is assigned to the thing that they're most excited to the thing that they're most excited to the thing that they're most excited about and that they're working in the about and that they're working in the about and that they're working in the best way possible. Now, what we do sort best way possible. Now, what we do sort best way possible. Now, what we do sort of solidify is when there's a product of solidify is when there's a product of solidify is when there's a product that has like legs. Like Cloud Design that has like legs. Like Cloud Design that has like legs. Like Cloud Design for example, started in this sort of ad for example, started in this sort of ad for example, started in this sort of ad hoc sort of grouped way and then now hoc sort of grouped way and then now hoc sort of grouped way and then now that like we've shipped it, it's gotten that like we've shipped it, it's gotten that like we've shipped it, it's gotten traction, we've done like a big second traction, we've done like a big second traction, we've done like a big second release in June. Like it's becoming like release in June. Like it's becoming like release in June. Like it's becoming like we've hired people for that specific we've hired people for that specific we've hired people for that specific team and it has more of a of a team and it has more of a of a team and it has more of a of a structure. So, it's like loose until it structure. So, it's like loose until it structure. So, it's like loose until it gets solidified down the line. gets solidified down the line. gets solidified down the line. >> What's the future of Cloud Design? I >> What's the future of Cloud Design? I >> What's the future of Cloud Design? I think a lot of people are very think a lot of people are very think a lot of people are very interested in It's one of your biggest interested in It's one of your biggest interested in It's one of your biggest launches this year. launches this year. launches this year. Where does this go? Where does this go? Where does this go? >> I think for me, I mean >> I think for me, I mean >> I think for me, I mean the things that are holding back Cloud the things that are holding back Cloud the things that are holding back Cloud Design from being even better is better Design from being even better is better Design from being even better is better interaction with our other surfaces. So, interaction with our other surfaces. So, interaction with our other surfaces. So, you know, I was designing something or I you know, I was designing something or I you know, I was designing something or I was talking to to Cloud Code the other was talking to to Cloud Code the other was talking to to Cloud Code the other day. I'm like, I want a really much more day. I'm like, I want a really much more day. I'm like, I want a really much more seamless like what I'm talking about the seamless like what I'm talking about the seamless like what I'm talking about the design for it, you know, and then after design for it, you know, and then after design for it, you know, and then after design back to that. I think in general design back to that. I think in general design back to that. I think in general it's I mean this goes back again to kind it's I mean this goes back again to kind it's I mean this goes back again to kind of unconstraining Cloud. Like the fact of unconstraining Cloud. Like the fact of unconstraining Cloud. Like the fact that our surfaces don't talk to each that our surfaces don't talk to each that our surfaces don't talk to each other as well as they could. I think other as well as they could. I think other as well as they could. I think really holds back a lot of interesting really holds back a lot of interesting really holds back a lot of interesting ideas around what we could do. So, I

ideas around what we could do. So, I ideas around what we could do. So, I think that's one like kind of major area think that's one like kind of major area think that's one like kind of major area that we're looking at. that we're looking at. that we're looking at. And then the other one is people like And then the other one is people like And then the other one is people like the lines between a Cloud Design and an the lines between a Cloud Design and an the lines between a Cloud Design and an app get blurry and blurrier over time. app get blurry and blurrier over time. app get blurry and blurrier over time. Like I've seen people Of course there's Like I've seen people Of course there's Like I've seen people Of course there's no like persistence but build like fully no like persistence but build like fully no like persistence but build like fully functional like even games which is functional like even games which is functional like even games which is definitely not what we designed Cloud definitely not what we designed Cloud definitely not what we designed Cloud Design for but you can do it. It's just Design for but you can do it. It's just Design for but you can do it. It's just HTML and JavaScript. HTML and JavaScript. HTML and JavaScript. So, blurring those lines even further So, blurring those lines even further So, blurring those lines even further and thinking through like what is the and thinking through like what is the and thinking through like what is the path from a like fully featured design path from a like fully featured design path from a like fully featured design that looks really well to really good to that looks really well to really good to that looks really well to really good to something that is maybe more like a something that is maybe more like a something that is maybe more like a artifact where you're actually able to artifact where you're actually able to artifact where you're actually able to go and you know, persist data and share go and you know, persist data and share go and you know, persist data and share it with others and build from there. So, it with others and build from there. So, it with others and build from there. So, I think that those lines get really I think that those lines get really I think that those lines get really interesting over time, too. interesting over time, too. interesting over time, too. >> Yeah. >> Yeah. >> Yeah. A big part of design is having taste. I A big part of design is having taste. I A big part of design is having taste. I actually asked Fable what Fable wants to actually asked Fable what Fable wants to actually asked Fable what Fable wants to ask you ask you ask you and this this this is what Fable came up and this this this is what Fable came up and this this this is what Fable came up with. You deleted almost all of Bourbon with. You deleted almost all of Bourbon with. You deleted almost all of Bourbon to get to Instagram which is like you to get to Instagram which is like you to get to Instagram which is like you had a whole you know solo mode whatever had a whole you know solo mode whatever had a whole you know solo mode whatever thing and you went to Instagram. What thing and you went to Instagram. What thing and you went to Instagram. What would you delete in AI or more spicy would you delete in AI or more spicy would you delete in AI or more spicy what would you delete in Claude? what would you delete in Claude? what would you delete in Claude? >> Oh, I like the spice.

>> Oh, I like the spice. >> Oh, I like the spice. I think I mean we have it's interesting I think I mean we have it's interesting I think I mean we have it's interesting we have a one of our slack channels is we have a one of our slack channels is we have a one of our slack channels is like project unship which is like what like project unship which is like what like project unship which is like what is in the product right now. It's is in the product right now. It's is in the product right now. It's I mean this is hard at Instagram. The I mean this is hard at Instagram. The I mean this is hard at Instagram. The Instagram we what Instagram we what Instagram we what things that had like four to five things that had like four to five things that had like four to five percent usage you're like oh that's percent usage you're like oh that's percent usage you're like oh that's really not very many but then you have really not very many but then you have really not very many but then you have like 20 features that each have four to like 20 features that each have four to like 20 features that each have four to five percent usage is like the classic five percent usage is like the classic five percent usage is like the classic Microsoft Word problem of like everybody Microsoft Word problem of like everybody Microsoft Word problem of like everybody uses some disjoint subset of the of the uses some disjoint subset of the of the uses some disjoint subset of the of the functionality. So that that's always the functionality. So that that's always the functionality. So that that's always the challenge. Now I think we're younger challenge. Now I think we're younger challenge. Now I think we're younger product so hopefully we have less of product so hopefully we have less of product so hopefully we have less of those things. Like we unship styles I those things. Like we unship styles I those things. Like we unship styles I think recently where it was like used by think recently where it was like used by think recently where it was like used by a small percentage of people and was not a small percentage of people and was not a small percentage of people and was not really AGI filled in a lot of ways it really AGI filled in a lot of ways it really AGI filled in a lot of ways it was like very sort of prescriptive in was like very sort of prescriptive in was like very sort of prescriptive in the way that it worked and skills very the way that it worked and skills very the way that it worked and skills very much better applications and then like much better applications and then like much better applications and then like that. So I think you have to be willing that. So I think you have to be willing that. So I think you have to be willing to take the primitives of like one to take the primitives of like one to take the primitives of like one generation of AI and like unship them or generation of AI and like unship them or generation of AI and like unship them or at least like supplement them or at least like supplement them or at least like supplement them or supplant them with the next one as well. supplant them with the next one as well. supplant them with the next one as well. I think the biggest thing is I look at I think the biggest thing is I look at I think the biggest thing is I look at it and I've been spending some time like it and I've been spending some time like it and I've been spending some time like outside labs on some of this is like man outside labs on some of this is like man outside labs on some of this is like man like we're asking people to make like like we're asking people to make like like we're asking people to make like code versus co-work versus like chat code versus co-work versus like chat code versus co-work versus like chat distinctions and like one they don't distinctions and like one they don't distinctions and like one they don't interoperate well and they can't interoperate well and they can't interoperate well and they can't delegate to each other and two I think delegate to each other and two I think delegate to each other and two I think the average person off the street could the average person off the street could the average person off the street could not explain to you why those are all not explain to you why those are all not explain to you why those are all different. So I think deleting some of different. So I think deleting some of different. So I think deleting some of the product complexity within our our the product complexity within our our the product complexity within our our code or our product I think is a a thing code or our product I think is a a thing code or our product I think is a a thing that would would serve well. Also that would would serve well. Also that would would serve well. Also because then Claude can do what it needs because then Claude can do what it needs because then Claude can do what it needs to do and and do well. Like there's to do and and do well. Like there's to do and and do well. Like there's nothing more frustrating than having a nothing more frustrating than having a nothing more frustrating than having a co-work session where you're like great co-work session where you're like great co-work session where you're like great I've mapped out exactly what I want you I've mapped out exactly what I want you I've mapped out exactly what I want you to build and then be like can you please to build and then be like can you please to build and then be like can you please like create a paragraph that I can paste

like create a paragraph that I can paste like create a paragraph that I can paste into Claude code? Like that is some 2020 into Claude code? Like that is some 2020 into Claude code? Like that is some 2020 you know kind of workflow there that you know kind of workflow there that you know kind of workflow there that really shouldn't exist anymore. really shouldn't exist anymore. really shouldn't exist anymore. >> Yeah. Um >> Yeah. Um >> Yeah. Um I think I think I think drawing lines on what you don't want to drawing lines on what you don't want to drawing lines on what you don't want to do and also sort of leaving room for do and also sort of leaving room for do and also sort of leaving room for others is interesting. Um a lot of others is interesting. Um a lot of others is interesting. Um a lot of people today is like the startups day people today is like the startups day people today is like the startups day for AI E or obviously very for AI E or obviously very for AI E or obviously very sympathetically aligned to startups. Uh sympathetically aligned to startups. Uh sympathetically aligned to startups. Uh but there's some anxiety in the room but there's some anxiety in the room but there's some anxiety in the room because tomorrow's Anthropic could wake because tomorrow's Anthropic could wake because tomorrow's Anthropic could wake up and publish up and publish up and publish some markdown files that destroy my some markdown files that destroy my some markdown files that destroy my industry. Um so industry. Um so industry. Um so >> [laughter] >> [laughter] >> [laughter] >> uh why should we not all just give up >> uh why should we not all just give up >> uh why should we not all just give up and join Anthropic? Like why bother and join Anthropic? Like why bother and join Anthropic? Like why bother starting any other company? starting any other company? starting any other company? >> Um I I mean actually joined one of the >> Um I I mean actually joined one of the >> Um I I mean actually joined one of the main reasons I joined Anthropic was main reasons I joined Anthropic was main reasons I joined Anthropic was because I saw how much this was like, because I saw how much this was like, because I saw how much this was like, you know, the models weren't that good you know, the models weren't that good you know, the models weren't that good at coding up but they were getting at coding up but they were getting at coding up but they were getting there. Like how much it would unlock there. Like how much it would unlock there. Like how much it would unlock like whole like next generation of like whole like next generation of like whole like next generation of startups. Not because it was going to startups. Not because it was going to startups. Not because it was going to solve their ideation or their taste, but solve their ideation or their taste, but solve their ideation or their taste, but because like it would make because like it would make because like it would make experimentation way simpler and and experimentation way simpler and and experimentation way simpler and and would get you to move faster. And I would get you to move faster. And I would get you to move faster. And I still like really believe that. And I still like really believe that. And I still like really believe that. And I mean it's the reality of, you know, mean it's the reality of, you know, mean it's the reality of, you know, uh uh uh And we saw this with like Instagram.

And we saw this with like Instagram. And we saw this with like Instagram. Like we would get questions from Like we would get questions from Like we would get questions from investors like, well, what happens when investors like, well, what happens when investors like, well, what happens when Google launches a photos product? It's Google launches a photos product? It's Google launches a photos product? It's like Google's going to launch a very like Google's going to launch a very like Google's going to launch a very googly photos product and it's going to googly photos product and it's going to googly photos product and it's going to have to be bound by the integrations have to be bound by the integrations have to be bound by the integrations that they already have and it's going to that they already have and it's going to that they already have and it's going to be like it's going to play to their be like it's going to play to their be like it's going to play to their strengths. And I think that is going to strengths. And I think that is going to strengths. And I think that is going to be true. And I'm not like giving advice be true. And I'm not like giving advice be true. And I'm not like giving advice on how to compete with Anthropic, I on how to compete with Anthropic, I on how to compete with Anthropic, I guess in a way, but like it's actually guess in a way, but like it's actually guess in a way, but like it's actually not because we're also a platform which not because we're also a platform which not because we're also a platform which is like there's so much I think room to is like there's so much I think room to is like there's so much I think room to be like laser obsessed with your be like laser obsessed with your be like laser obsessed with your particular vertical or your industry or particular vertical or your industry or particular vertical or your industry or group of people that you know really group of people that you know really group of people that you know really well in a way that like none of the labs well in a way that like none of the labs well in a way that like none of the labs are ever going to get to that level of are ever going to get to that level of are ever going to get to that level of uh of understanding and like therefore uh of understanding and like therefore uh of understanding and like therefore get that kind of adoption and user love get that kind of adoption and user love get that kind of adoption and user love and and build that out. Now, it's and and build that out. Now, it's and and build that out. Now, it's definitely harder in the age where like definitely harder in the age where like definitely harder in the age where like the models can just do a lot and so the models can just do a lot and so the models can just do a lot and so there's, you know, some of these things there's, you know, some of these things there's, you know, some of these things can be like skillified and like maybe can be like skillified and like maybe can be like skillified and like maybe don't need their own dedicated product. don't need their own dedicated product. don't need their own dedicated product. But I think it's like the hard stuff is But I think it's like the hard stuff is But I think it's like the hard stuff is still hard. It's like understanding the still hard. It's like understanding the still hard. It's like understanding the needs of people, like figuring out how needs of people, like figuring out how needs of people, like figuring out how you're going to reach them, uh listening you're going to reach them, uh listening you're going to reach them, uh listening to them and iterating on them really to them and iterating on them really to them and iterating on them really quickly. Like quickly. Like quickly. Like it is still the case that like a group it is still the case that like a group it is still the case that like a group of four or five people obsessed with a of four or five people obsessed with a of four or five people obsessed with a problem is going to move faster than problem is going to move faster than problem is going to move faster than those same people at any other kind of those same people at any other kind of those same people at any other kind of organization that are like, you know, organization that are like, you know, organization that are like, you know, subject just to the complexity. I just subject just to the complexity. I just subject just to the complexity. I just mentioned the like the fact that we mentioned the like the fact that we mentioned the like the fact that we have, you know, a lot of different have, you know, a lot of different have, you know, a lot of different products that kind of interoperate. Like products that kind of interoperate. Like products that kind of interoperate. Like that's a interesting constraint that we that's a interesting constraint that we that's a interesting constraint that we have to work through. It's an advantage have to work through. It's an advantage have to work through. It's an advantage in other ways, right? So, yeah, I'm in other ways, right? So, yeah, I'm in other ways, right? So, yeah, I'm still like very long and bullish on still like very long and bullish on still like very long and bullish on startups and um it's just startups and um it's just startups and um it's just it tapers over the fact that like it tapers over the fact that like it tapers over the fact that like writing code was never the like the writing code was never the like the writing code was never the like the limiting part. You know, maybe it was on limiting part. You know, maybe it was on limiting part. You know, maybe it was on the timeline perspective, but it was the timeline perspective, but it was the timeline perspective, but it was never like the thing that was going to never like the thing that was going to never like the thing that was going to like make or break a startup. It's like make or break a startup. It's like make or break a startup. It's really that space and user really that space and user really that space and user understanding.

understanding. understanding. >> Yeah. Uh domain knowledge. >> Yeah. Uh domain knowledge. >> Yeah. Uh domain knowledge. >> Yeah. >> Yeah. >> Yeah. >> Uh today is also our day for vertical >> Uh today is also our day for vertical >> Uh today is also our day for vertical AI. Uh one of our AI. Uh one of our AI. Uh one of our uh returning speakers and top speakers, uh returning speakers and top speakers, uh returning speakers and top speakers, Chris Lovejoy, uh was always talking Chris Lovejoy, uh was always talking Chris Lovejoy, uh was always talking about vertical AI. He was in from about vertical AI. He was in from about vertical AI. He was in from interior in the healthcare space. And interior in the healthcare space. And interior in the healthcare space. And then recently I was I invited him back then recently I was I invited him back then recently I was I invited him back and turned out he you guys just hired and turned out he you guys just hired and turned out he you guys just hired him for your uh healthcare efforts. Um him for your uh healthcare efforts. Um him for your uh healthcare efforts. Um we also our next big one is also we also our next big one is also we also our next big one is also finance. You know, we have a yeah, a finance. You know, we have a yeah, a finance. You know, we have a yeah, a finance track. You guys just had a huge finance track. You guys just had a huge finance track. You guys just had a huge finance event in New York City. Um and finance event in New York City. Um and finance event in New York City. Um and where our next uh AI is is sort of where our next uh AI is is sort of where our next uh AI is is sort of finance focused. What are you seeing finance focused. What are you seeing finance focused. What are you seeing there? Any you know, any potential uh there? Any you know, any potential uh there? Any you know, any potential uh for Claude? Obviously a lot of Excel for Claude? Obviously a lot of Excel for Claude? Obviously a lot of Excel Excel spreadsheets. Excel spreadsheets. Excel spreadsheets. >> Yeah. No, I think that there's there's a >> Yeah. No, I think that there's there's a >> Yeah. No, I think that there's there's a lot in there, too. And that's like an lot in there, too. And that's like an lot in there, too. And that's like an area where uh you could see the model area where uh you could see the model area where uh you could see the model get clearly better at it like sort of get clearly better at it like sort of get clearly better at it like sort of generation to generation. And there's, generation to generation. And there's, generation to generation. And there's, you know, there's some good sort of you know, there's some good sort of you know, there's some good sort of vertical specific uh uh finance startups vertical specific uh uh finance startups vertical specific uh uh finance startups that have like done their own um that have like done their own um that have like done their own um evals, which has also been interesting evals, which has also been interesting evals, which has also been interesting to to track. And it's not like we're to to track. And it's not like we're to to track. And it's not like we're like sort of playing to the eval, but it like sort of playing to the eval, but it like sort of playing to the eval, but it is a useful sort of barometer around is a useful sort of barometer around is a useful sort of barometer around like is this actually getting better um like is this actually getting better um like is this actually getting better um at these finance use cases. I think the at these finance use cases. I think the at these finance use cases. I think the interesting blend that's going to happen interesting blend that's going to happen interesting blend that's going to happen um is this mix of, again, the model um is this mix of, again, the model um is this mix of, again, the model having the flexibility to like dive in having the flexibility to like dive in having the flexibility to like dive in and create just-in-time analyses or and create just-in-time analyses or and create just-in-time analyses or dashboards or workflows with like some dashboards or workflows with like some dashboards or workflows with like some sense of like what is the not immutable, sense of like what is the not immutable, sense of like what is the not immutable, but at least like verified sort of set but at least like verified sort of set but at least like verified sort of set of data. And so like uh of data. And so like uh of data. And so like uh set having all of that be totally free set having all of that be totally free set having all of that be totally free form, I think is a recipe for confusion form, I think is a recipe for confusion form, I think is a recipe for confusion and is like not what most companies in and is like not what most companies in and is like not what most companies in the financial services space want. So, the financial services space want. So, the financial services space want. So, finding that right uh sort of cut line

finding that right uh sort of cut line finding that right uh sort of cut line where you have verifiability and audit where you have verifiability and audit where you have verifiability and audit logging and and sort of data provenance logging and and sort of data provenance logging and and sort of data provenance here, but not in a way that constrains here, but not in a way that constrains here, but not in a way that constrains the kinds of applications that you can the kinds of applications that you can the kinds of applications that you can build on top, I think is a lot of the build on top, I think is a lot of the build on top, I think is a lot of the art that we're seeing in that space as art that we're seeing in that space as art that we're seeing in that space as well. Um and I think, you know, if you well. Um and I think, you know, if you well. Um and I think, you know, if you solve it well, you can you can get the solve it well, you can you can get the solve it well, you can you can get the best of both worlds. The hard part is a best of both worlds. The hard part is a best of both worlds. The hard part is a lot of the systems that were built to do lot of the systems that were built to do lot of the systems that were built to do the verifiability audibility like are the verifiability audibility like are the verifiability audibility like are kind of almost by design not super kind of almost by design not super kind of almost by design not super flexible in terms of agentic workloads flexible in terms of agentic workloads flexible in terms of agentic workloads on top. So, I think there's opportunity on top. So, I think there's opportunity on top. So, I think there's opportunity at both sides of the stack there. at both sides of the stack there. at both sides of the stack there. >> Yeah. Um I think I I also agree we'll be >> Yeah. Um I think I I also agree we'll be >> Yeah. Um I think I I also agree we'll be exploring that in in New York. Um the exploring that in in New York. Um the exploring that in in New York. Um the last thing I want to end on is on mental last thing I want to end on is on mental last thing I want to end on is on mental health, which we don't talk about enough health, which we don't talk about enough health, which we don't talk about enough in technical conference conferences. Um in technical conference conferences. Um in technical conference conferences. Um you've seen a lot of hyper growth. you've seen a lot of hyper growth. you've seen a lot of hyper growth. People are just always refreshing their People are just always refreshing their People are just always refreshing their timelines and it's exhausting. timelines and it's exhausting. timelines and it's exhausting. Um Um Um how do you advise people who are working how do you advise people who are working how do you advise people who are working 996 to avoid burnout? 996 to avoid burnout? 996 to avoid burnout? >> Yeah. I mean, I think this is a hard >> Yeah. I mean, I think this is a hard >> Yeah. I mean, I think this is a hard one. I mean, and one. I mean, and one. I mean, and it is, I'm sure you are experiencing it is, I'm sure you are experiencing it is, I'm sure you are experiencing this cuz you're all working in this this cuz you're all working in this this cuz you're all working in this industry like industry like industry like it is it is it is you know, multiples more intense and you know, multiples more intense and you know, multiples more intense and things move much more quickly at an things move much more quickly at an things move much more quickly at an Instagram like our the two things that Instagram like our the two things that Instagram like our the two things that we were thinking about was like, what is we were thinking about was like, what is we were thinking about was like, what is Apple going to announce at WWDC and is Apple going to announce at WWDC and is Apple going to announce at WWDC and is it going to like totally mess us up or it going to like totally mess us up or it going to like totally mess us up or boost us, right? So, that's like once a boost us, right? So, that's like once a boost us, right? So, that's like once a year. Um or, you know, maybe a year. Um or, you know, maybe a year. Um or, you know, maybe a competitor launches every three or four competitor launches every three or four competitor launches every three or four months, right? And uh it is definitely months, right? And uh it is definitely months, right? And uh it is definitely not that. It's not that. It's not that. It's a topic we we do a when we do do our a topic we we do a when we do do our a topic we we do a when we do do our weekly all hands. Usually on Wednesdays weekly all hands. Usually on Wednesdays weekly all hands. Usually on Wednesdays and we have a slide that's like the week and we have a slide that's like the week and we have a slide that's like the week in AI at Pinterest is and it's only in AI at Pinterest is and it's only in AI at Pinterest is and it's only Wednesday and like and inevitably like Wednesday and like and inevitably like Wednesday and like and inevitably like some competitor has shipped a new model some competitor has shipped a new model some competitor has shipped a new model and like there's been a like new product

and like there's been a like new product and like there's been a like new product and maybe there's some interesting thing and maybe there's some interesting thing and maybe there's some interesting thing happening happening happening um uh on the regulation side. Like if um uh on the regulation side. Like if um uh on the regulation side. Like if things are moving really, really things are moving really, really things are moving really, really quickly. I think the way I try to stay quickly. I think the way I try to stay quickly. I think the way I try to stay at least relatively sane, um one is like at least relatively sane, um one is like at least relatively sane, um one is like actually carving time off and I think actually carving time off and I think actually carving time off and I think the topic co-founders do a good job of the topic co-founders do a good job of the topic co-founders do a good job of like saying like, look, like burnout if like saying like, look, like burnout if like saying like, look, like burnout if you you out, like you're kind of done. you you out, like you're kind of done. you you out, like you're kind of done. I've seen it happen unfortunately to I've seen it happen unfortunately to I've seen it happen unfortunately to people I'm really close to and then it people I'm really close to and then it people I'm really close to and then it takes a long time to recover from that. takes a long time to recover from that. takes a long time to recover from that. Um so actually encouraging people like Um so actually encouraging people like Um so actually encouraging people like there's no job that is so important that there's no job that is so important that there's no job that is so important that you can't be offline for a couple of you can't be offline for a couple of you can't be offline for a couple of days. Um so I think that's like a big days. Um so I think that's like a big days. Um so I think that's like a big key like piece in there. So like let's key like piece in there. So like let's key like piece in there. So like let's >> [applause] >> [applause] >> [applause] >> strongly believe. >> strongly believe. >> strongly believe. Um Um Um And if it is you're probably doing And if it is you're probably doing And if it is you're probably doing something wrong and you talk to somebody something wrong and you talk to somebody something wrong and you talk to somebody who could be a mentor to figure out how who could be a mentor to figure out how who could be a mentor to figure out how you can unblock that. Um and then I you can unblock that. Um and then I you can unblock that. Um and then I think the other one as well is like I think the other one as well is like I think the other one as well is like I love sports and like uh love sports and like uh love sports and like uh this is the notion of like you're never this is the notion of like you're never this is the notion of like you're never as good as like your best game and as good as like your best game and as good as like your best game and you're never as bad as your worst game. you're never as bad as your worst game. you're never as bad as your worst game. I think that's also really true. Like I I think that's also really true. Like I I think that's also really true. Like I know like in AI there's like the you know like in AI there's like the you know like in AI there's like the you know it's so over we're so back thing. know it's so over we're so back thing. know it's so over we're so back thing. Like that like if you internalize that Like that like if you internalize that Like that like if you internalize that that cycle is always going to be at play that cycle is always going to be at play that cycle is always going to be at play in some way, you realize like it's never in some way, you realize like it's never in some way, you realize like it's never that bad. Like Ben Horowitz's book is that bad. Like Ben Horowitz's book is that bad. Like Ben Horowitz's book is the hard thing about hard things has the hard thing about hard things has the hard thing about hard things has this chapter on like we're effed it's this chapter on like we're effed it's this chapter on like we're effed it's over and like that feeling as a startup over and like that feeling as a startup over and like that feeling as a startup that probably many of you have had at that probably many of you have had at that probably many of you have had at startups where you're like oh I can't startups where you're like oh I can't startups where you're like oh I can't believe this thing happened like we're believe this thing happened like we're believe this thing happened like we're never going to like recover from this. I never going to like recover from this. I never going to like recover from this. I definitely we definitely had an definitely we definitely had an definitely we definitely had an Instagram a couple of times. And then Instagram a couple of times. And then Instagram a couple of times. And then you get through it and like it's that you get through it and like it's that you get through it and like it's that like def like defines the company when like def like defines the company when like def like defines the company when you can actually go through that. I try you can actually go through that. I try you can actually go through that. I try to remind myself and the team here even to remind myself and the team here even to remind myself and the team here even with an Entropic which is like like this with an Entropic which is like like this with an Entropic which is like like this is a is a fast-moving but is also a long

is a is a fast-moving but is also a long is a is a fast-moving but is also a long game. And it's like we're never it's game. And it's like we're never it's game. And it's like we're never it's never just about today's model launch never just about today's model launch never just about today's model launch and reaction or this product launch and and reaction or this product launch and and reaction or this product launch and everything else. Like you're playing and everything else. Like you're playing and everything else. Like you're playing and you're building and you just have to you're building and you just have to you're building and you just have to trust that you're building like the team trust that you're building like the team trust that you're building like the team and culture that is going to get through and culture that is going to get through and culture that is going to get through those things and have that sense of those things and have that sense of those things and have that sense of perspective even if perspective is perspective even if perspective is perspective even if perspective is saying like look 3 months ago we were in saying like look 3 months ago we were in saying like look 3 months ago we were in a similar position. Maybe it's not a a similar position. Maybe it's not a a similar position. Maybe it's not a year it's just a matter of months but year it's just a matter of months but year it's just a matter of months but it's still like zooming out and not it's still like zooming out and not it's still like zooming out and not taking things not letting your internal taking things not letting your internal taking things not letting your internal sort of like sense of self and success sort of like sense of self and success sort of like sense of self and success be so driven by the day-to-day. be so driven by the day-to-day. be so driven by the day-to-day. >> Yeah. It has anyone any coach or mentor >> Yeah. It has anyone any coach or mentor >> Yeah. It has anyone any coach or mentor said something to you that you repeat to said something to you that you repeat to said something to you that you repeat to yourself that gets you through the hard yourself that gets you through the hard yourself that gets you through the hard tough times? tough times? tough times? >> Um I think the biggest one was >> Um I think the biggest one was >> Um I think the biggest one was this like sense of like if you're this like sense of like if you're this like sense of like if you're feeling something it's really often the feeling something it's really often the feeling something it's really often the case that other people on the team are case that other people on the team are case that other people on the team are feeling it too. So this is like advice I feeling it too. So this is like advice I feeling it too. So this is like advice I got from my my coach around just being got from my my coach around just being got from my my coach around just being like like just verbalizing emotions like like like just verbalizing emotions like like like just verbalizing emotions like even saying like hey I'm feeling really even saying like hey I'm feeling really even saying like hey I'm feeling really stressed out about this or yeah I'm stressed out about this or yeah I'm stressed out about this or yeah I'm really sad that we are shutting down really sad that we are shutting down really sad that we are shutting down this labs initiative. I literally had this labs initiative. I literally had this labs initiative. I literally had this meeting a couple months ago where I this meeting a couple months ago where I this meeting a couple months ago where I was working really hard on something and was working really hard on something and was working really hard on something and I kicked off the meeting like I kicked off the meeting like I kicked off the meeting like I'll kick it off like I'm really sad I'll kick it off like I'm really sad I'll kick it off like I'm really sad like I'm frustrated like I wish this like I'm frustrated like I wish this like I'm frustrated like I wish this thing had worked out and I think that thing had worked out and I think that thing had worked out and I think that holds the space for other people to be holds the space for other people to be holds the space for other people to be like yeah I'm pissed off too or like I'm like yeah I'm pissed off too or like I'm like yeah I'm pissed off too or like I'm sad too and like I think giving that sad too and like I think giving that sad too and like I think giving that advice around like not advice around like not advice around like not I think if you can get yourself to be I think if you can get yourself to be I think if you can get yourself to be open and vulnerable it often like lets open and vulnerable it often like lets open and vulnerable it often like lets other people verbalize that and then you other people verbalize that and then you other people verbalize that and then you can from there you can be like great can from there you can be like great can from there you can be like great what are we going to do about it like what are we going to do about it like what are we going to do about it like you know it's much easier to start from you know it's much easier to start from you know it's much easier to start from that place.

that place. that place. >> Yeah we actually kicked off AIE with a >> Yeah we actually kicked off AIE with a >> Yeah we actually kicked off AIE with a session from Carol Robbins who runs session from Carol Robbins who runs session from Carol Robbins who runs touchy feely at Stanford touchy feely at Stanford touchy feely at Stanford and I can't think of a better way to end and I can't think of a better way to end and I can't think of a better way to end than encouraging people to talk about than encouraging people to talk about than encouraging people to talk about their feelings manage their mental their feelings manage their mental their feelings manage their mental health and keep shipping. health and keep shipping. health and keep shipping. >> Yeah. Thanks so much Mike. >> Yeah. Thanks so much Mike. >> Yeah. Thanks so much Mike. >> Thanks for having me.