Dario Amodei: Anthropic CEO on Claude, AGI & the Future of AI & Humanity | Lex Fridman Podcast #452

if you extrapolate the curves that we've if you extrapolate the curves that we've had so far right if if you say well I had so far right if if you say well I had so far right if if you say well I don't know we're starting to get to like don't know we're starting to get to like don't know we're starting to get to like PhD level and and last year we were at PhD level and and last year we were at PhD level and and last year we were at undergraduate level and the year before undergraduate level and the year before undergraduate level and the year before we were at like the level of a high we were at like the level of a high we were at like the level of a high school student again you can you can school student again you can you can school student again you can you can quibble with at what tasks and for what quibble with at what tasks and for what quibble with at what tasks and for what we're still missing modalities but those we're still missing modalities but those we're still missing modalities but those are being added like computer use was are being added like computer use was are being added like computer use was added like image generation has been added like image generation has been added like image generation has been added if you just kind of like eyeball added if you just kind of like eyeball added if you just kind of like eyeball the rate at which these capabilities are the rate at which these capabilities are the rate at which these capabilities are increasing it does make you think that increasing it does make you think that increasing it does make you think that we'll get there by 2026 or 2027 I think we'll get there by 2026 or 2027 I think we'll get there by 2026 or 2027 I think there are still worlds where it doesn't there are still worlds where it doesn't there are still worlds where it doesn't happen in in a 100 years those world the happen in in a 100 years those world the happen in in a 100 years those world the number of those worlds is rapidly number of those worlds is rapidly number of those worlds is rapidly decreasing we are rapidly running out of decreasing we are rapidly running out of decreasing we are rapidly running out of truly convincing blockers truly truly convincing blockers truly truly convincing blockers truly compelling reasons why this will not compelling reasons why this will not compelling reasons why this will not happen in the next few years the scale happen in the next few years the scale happen in the next few years the scale up is very quick like we we do this up is very quick like we we do this up is very quick like we we do this today we make a model and then we deploy today we make a model and then we deploy today we make a model and then we deploy thousands maybe tens of thousands of thousands maybe tens of thousands of thousands maybe tens of thousands of instances of it I think by the time you instances of it I think by the time you instances of it I think by the time you know certainly within two to three years know certainly within two to three years know certainly within two to three years whether we have these super powerful AIS whether we have these super powerful AIS whether we have these super powerful AIS or not ERS are going to get to the size or not ERS are going to get to the size or not ERS are going to get to the size where you'll be able to deploy millions where you'll be able to deploy millions where you'll be able to deploy millions of these I am optimistic about meaning I of these I am optimistic about meaning I of these I am optimistic about meaning I worry about economics and the worry about economics and the worry about economics and the concentration of power that's actually concentration of power that's actually concentration of power that's actually what I worry about more the abuse of what I worry about more the abuse of what I worry about more the abuse of power and AI increases the amount of power and AI increases the amount of power and AI increases the amount of power in the world and if you power in the world and if you power in the world and if you concentrate that power and abuse that concentrate that power and abuse that concentrate that power and abuse that power it can do immeasurable damage yes power it can do immeasurable damage yes power it can do immeasurable damage yes it's very frightening it's very it's it's very frightening it's very it's it's very frightening it's very it's very very very frightening the following is a frightening the following is a frightening the following is a conversation with Dario amade CEO of conversation with Dario amade CEO of conversation with Dario amade CEO of anthropic the company that created

anthropic the company that created anthropic the company that created Claude that is currently and often at Claude that is currently and often at Claude that is currently and often at the top of most llm Benchmark leader the top of most llm Benchmark leader the top of most llm Benchmark leader boards on top of that Dario and the boards on top of that Dario and the boards on top of that Dario and the anthropic team have been outspoken anthropic team have been outspoken anthropic team have been outspoken advocates for taking the topic of AI advocates for taking the topic of AI advocates for taking the topic of AI safety very seriously and they have safety very seriously and they have safety very seriously and they have continued to publish a lot of continued to publish a lot of continued to publish a lot of fascinating AI research on this and fascinating AI research on this and fascinating AI research on this and other topics I'm also joined afterwards other topics I'm also joined afterwards other topics I'm also joined afterwards by two other brilliant people from by two other brilliant people from by two other brilliant people from propic first Amanda ascal who is a propic first Amanda ascal who is a propic first Amanda ascal who is a researcher working on alignment and researcher working on alignment and researcher working on alignment and fine-tuning of Claude including the fine-tuning of Claude including the fine-tuning of Claude including the design of claude's character and design of claude's character and design of claude's character and personality a few folks told me she has personality a few folks told me she has personality a few folks told me she has probably talked with Claude more than probably talked with Claude more than probably talked with Claude more than any human at anthropic so she was any human at anthropic so she was any human at anthropic so she was definitely a fascinating person to talk definitely a fascinating person to talk definitely a fascinating person to talk to about prompt engineering and to about prompt engineering and to about prompt engineering and practical advice on how to get the best practical advice on how to get the best practical advice on how to get the best out of Claude after that chrisa stopped out of Claude after that chrisa stopped out of Claude after that chrisa stopped by for chat he's one of the pioneers of by for chat he's one of the pioneers of by for chat he's one of the pioneers of the field of mechanistic the field of mechanistic the field of mechanistic interpretability which is an exciting interpretability which is an exciting interpretability which is an exciting set of efforts that aims to reverse set of efforts that aims to reverse set of efforts that aims to reverse engineer neural networks to figure out engineer neural networks to figure out engineer neural networks to figure out what's going on inside inferring what's going on inside inferring what's going on inside inferring behaviors from neural activation behaviors from neural activation behaviors from neural activation patterns inside the network this is a patterns inside the network this is a patterns inside the network this is a very promising approach for keeping very promising approach for keeping very promising approach for keeping future super intelligent AI systems safe future super intelligent AI systems safe future super intelligent AI systems safe for example by detecting from the for example by detecting from the for example by detecting from the activations when the model is trying to activations when the model is trying to activations when the model is trying to deceive the human it is talking deceive the human it is talking deceive the human it is talking to this is Alex Freedman podcast to

to this is Alex Freedman podcast to to this is Alex Freedman podcast to support it please check out our sponsors support it please check out our sponsors support it please check out our sponsors in the description and now dear friends in the description and now dear friends in the description and now dear friends here's Dario here's Dario here's Dario amade let's start with a big idea of amade let's start with a big idea of amade let's start with a big idea of scaling laws and the scaling hypothesis scaling laws and the scaling hypothesis scaling laws and the scaling hypothesis what is it what is its history and where what is it what is its history and where what is it what is its history and where do we stand today so I can only describe do we stand today so I can only describe do we stand today so I can only describe it as it you know as it relates to kind it as it you know as it relates to kind it as it you know as it relates to kind of my own experience but I've been in of my own experience but I've been in of my own experience but I've been in the AI field for about uh 10 years and the AI field for about uh 10 years and the AI field for about uh 10 years and it was something I noticed very early on it was something I noticed very early on it was something I noticed very early on so I first joined the AI world when I so I first joined the AI world when I so I first joined the AI world when I was uh working at BYU with Andrew in in was uh working at BYU with Andrew in in was uh working at BYU with Andrew in in late 2014 which is almost exactly 10 late 2014 which is almost exactly 10 late 2014 which is almost exactly 10 years ago now and the first thing we years ago now and the first thing we years ago now and the first thing we worked on was speech recognition systems worked on was speech recognition systems worked on was speech recognition systems and in those days I think deep learning and in those days I think deep learning and in those days I think deep learning was a new thing it had made lots of was a new thing it had made lots of was a new thing it had made lots of progress but everyone was always saying progress but everyone was always saying progress but everyone was always saying we don't have the algorithms we need to we don't have the algorithms we need to we don't have the algorithms we need to succeed you know we we we we're we're succeed you know we we we we're we're succeed you know we we we we're we're not we're only matching a tiny tiny not we're only matching a tiny tiny not we're only matching a tiny tiny fraction there's so much we need to kind fraction there's so much we need to kind fraction there's so much we need to kind of discover algorithmically we haven't of discover algorithmically we haven't of discover algorithmically we haven't found the picture of how to match the found the picture of how to match the found the picture of how to match the human brain uh and when you know in some human brain uh and when you know in some human brain uh and when you know in some ways was fortunate I was kind of you ways was fortunate I was kind of you ways was fortunate I was kind of you know you can have almost beginner's luck know you can have almost beginner's luck know you can have almost beginner's luck right I was like a a newcomer to the right I was like a a newcomer to the right I was like a a newcomer to the field and you know I looked at the field and you know I looked at the field and you know I looked at the neural net that we were using for speech neural net that we were using for speech neural net that we were using for speech the recurrent neural networks and I said the recurrent neural networks and I said the recurrent neural networks and I said I don't know what if you make them I don't know what if you make them I don't know what if you make them bigger and give them more layers and bigger and give them more layers and bigger and give them more layers and what if you scale up the data along with what if you scale up the data along with what if you scale up the data along with this right I just saw these as as like this right I just saw these as as like this right I just saw these as as like independent dials that you could turn independent dials that you could turn independent dials that you could turn and I noticed that the model started to and I noticed that the model started to and I noticed that the model started to do better and better as you gave them do better and better as you gave them do better and better as you gave them more data as you as you made the models more data as you as you made the models more data as you as you made the models larger as you trained them for longer um

larger as you trained them for longer um larger as you trained them for longer um and I I didn't measure things precisely and I I didn't measure things precisely and I I didn't measure things precisely in those days but but along with with in those days but but along with with in those days but but along with with colleagues we very much got the informal colleagues we very much got the informal colleagues we very much got the informal sense that the more data and the more sense that the more data and the more sense that the more data and the more compute and the more training you put compute and the more training you put compute and the more training you put into these models the better they into these models the better they into these models the better they perform and so initially my thinking was perform and so initially my thinking was perform and so initially my thinking was hey maybe that is just true for speech hey maybe that is just true for speech hey maybe that is just true for speech recognition systems right maybe maybe recognition systems right maybe maybe recognition systems right maybe maybe that's just one particular quirk one that's just one particular quirk one that's just one particular quirk one particular area I think it wasn't until particular area I think it wasn't until particular area I think it wasn't until 2017 when I first saw the results from 2017 when I first saw the results from 2017 when I first saw the results from gpt1 that it clicked for me that gpt1 that it clicked for me that gpt1 that it clicked for me that language is probably the area in which language is probably the area in which language is probably the area in which we can do this we can get trillions of we can do this we can get trillions of we can do this we can get trillions of words of language data we can train on words of language data we can train on words of language data we can train on them and the models we were training in them and the models we were training in them and the models we were training in those days were tiny you could train those days were tiny you could train those days were tiny you could train them on one to eight gpus whereas you them on one to eight gpus whereas you them on one to eight gpus whereas you know now we train jobs on tens of know now we train jobs on tens of know now we train jobs on tens of thousands soon going to hundreds of thousands soon going to hundreds of thousands soon going to hundreds of thousands of gpus and so when I when I thousands of gpus and so when I when I thousands of gpus and so when I when I saw those two things together um and you saw those two things together um and you saw those two things together um and you know there were a few people like ilaser know there were a few people like ilaser know there were a few people like ilaser who who you've interviewed who had who who you've interviewed who had who who you've interviewed who had somewhat similar reviews right he might somewhat similar reviews right he might somewhat similar reviews right he might have been the first one although I think have been the first one although I think have been the first one although I think a few people came to came to similar a few people came to came to similar a few people came to came to similar views around the same time Right There views around the same time Right There views around the same time Right There Was You Know Rich Sutton's bitter lesson Was You Know Rich Sutton's bitter lesson Was You Know Rich Sutton's bitter lesson there was gur wrote about the scaling there was gur wrote about the scaling there was gur wrote about the scaling hypothesis but I think somewhere between hypothesis but I think somewhere between hypothesis but I think somewhere between 2014 and 2017 was when it really clicked 2014 and 2017 was when it really clicked 2014 and 2017 was when it really clicked for me when I really got conviction that for me when I really got conviction that for me when I really got conviction that hey we're going to be able to do these hey we're going to be able to do these hey we're going to be able to do these incredibly wide cognitive tasks if we incredibly wide cognitive tasks if we incredibly wide cognitive tasks if we just if we just scale up the models and just if we just scale up the models and just if we just scale up the models and at at every stage of scaling there are at at every stage of scaling there are at at every stage of scaling there are always arguments and you know when I

always arguments and you know when I always arguments and you know when I first heard them honestly I thought first heard them honestly I thought first heard them honestly I thought probably I'm the one who's wrong and you probably I'm the one who's wrong and you probably I'm the one who's wrong and you know all these all these experts in the know all these all these experts in the know all these all these experts in the field are right they know the situation field are right they know the situation field are right they know the situation better better than I do right there's better better than I do right there's better better than I do right there's you know the Chomsky argument about like you know the Chomsky argument about like you know the Chomsky argument about like you can get syntactics but you can't get you can get syntactics but you can't get you can get syntactics but you can't get semantics there's this idea oh you can semantics there's this idea oh you can semantics there's this idea oh you can make a sentence make sense but you can't make a sentence make sense but you can't make a sentence make sense but you can't make a paragraph makes sense the latest make a paragraph makes sense the latest make a paragraph makes sense the latest one we have today is uh you know we're one we have today is uh you know we're one we have today is uh you know we're going to run out of data or the data going to run out of data or the data going to run out of data or the data isn't high quality enough or models isn't high quality enough or models isn't high quality enough or models can't reason and and each time every can't reason and and each time every can't reason and and each time every time we manage to we manage to either time we manage to we manage to either time we manage to we manage to either find a way around or scaling just is the find a way around or scaling just is the find a way around or scaling just is the way around um sometimes it's one way around um sometimes it's one way around um sometimes it's one sometimes it's the other uh and and so sometimes it's the other uh and and so sometimes it's the other uh and and so I'm now at this point I I I still think I'm now at this point I I I still think I'm now at this point I I I still think you know it's it's it's always quite you know it's it's it's always quite you know it's it's it's always quite uncertain we have nothing but inductive uncertain we have nothing but inductive uncertain we have nothing but inductive inference to tell us that the next few inference to tell us that the next few inference to tell us that the next few years are going to be like the next the years are going to be like the next the years are going to be like the next the last 10 years but but I've seen I've last 10 years but but I've seen I've last 10 years but but I've seen I've seen the movie enough times I've seen seen the movie enough times I've seen seen the movie enough times I've seen the story happen for for enough times to the story happen for for enough times to the story happen for for enough times to to really believe that probably the to really believe that probably the to really believe that probably the scaling is going to continue and that scaling is going to continue and that scaling is going to continue and that there's some magic to it that we haven't there's some magic to it that we haven't there's some magic to it that we haven't really explained on a theoretical basis really explained on a theoretical basis really explained on a theoretical basis yet and of course the scaling here is yet and of course the scaling here is yet and of course the scaling here is bigger networks bigger data bigger bigger networks bigger data bigger bigger networks bigger data bigger compute yes all in in particular linear compute yes all in in particular linear compute yes all in in particular linear scaling up of bigger networks bigger scaling up of bigger networks bigger scaling up of bigger networks bigger training times and uh more and and more training times and uh more and and more training times and uh more and and more data uh so all of these things almost data uh so all of these things almost data uh so all of these things almost like a chemical reaction you know you like a chemical reaction you know you like a chemical reaction you know you have three ingredients in the chemical have three ingredients in the chemical have three ingredients in the chemical reaction and you need to linearly scale reaction and you need to linearly scale reaction and you need to linearly scale up the three ingredients if you scale up up the three ingredients if you scale up up the three ingredients if you scale up one not the others you run out of the one not the others you run out of the one not the others you run out of the other reagents and and the reaction

other reagents and and the reaction other reagents and and the reaction stops but if you scale up everything stops but if you scale up everything stops but if you scale up everything everything in series then then the everything in series then then the everything in series then then the reaction can proceed and of course now reaction can proceed and of course now reaction can proceed and of course now that you have this kind of empirical that you have this kind of empirical that you have this kind of empirical scienceart you can apply it to scienceart you can apply it to scienceart you can apply it to other uh more nuanced things like other uh more nuanced things like other uh more nuanced things like scaling laws applied to interpretability scaling laws applied to interpretability scaling laws applied to interpretability or scaling laws applied to posttraining or scaling laws applied to posttraining or scaling laws applied to posttraining or just seeing how does this thing scale or just seeing how does this thing scale or just seeing how does this thing scale but the big scaling law I guess the but the big scaling law I guess the but the big scaling law I guess the underlying scaling hypothesis has to do underlying scaling hypothesis has to do underlying scaling hypothesis has to do with big networks Big Data leads to with big networks Big Data leads to with big networks Big Data leads to intelligence yeah we've we've documented intelligence yeah we've we've documented intelligence yeah we've we've documented scaling laws in lots of domains other scaling laws in lots of domains other scaling laws in lots of domains other than language right so uh initially the than language right so uh initially the than language right so uh initially the the paper we did that first showed it the paper we did that first showed it the paper we did that first showed it was in early 2020 where we first showed was in early 2020 where we first showed was in early 2020 where we first showed it for language there was then some work it for language there was then some work it for language there was then some work late in 2020 where we showed the same late in 2020 where we showed the same late in 2020 where we showed the same thing for other modalities like images thing for other modalities like images thing for other modalities like images video video video text to image image to text math they text to image image to text math they text to image image to text math they all had the same pattern and and you're all had the same pattern and and you're all had the same pattern and and you're right now there are other stages like right now there are other stages like right now there are other stages like posttraining or there are new types of posttraining or there are new types of posttraining or there are new types of reasoning models and in in in all of reasoning models and in in in all of reasoning models and in in in all of those cases that we've measured we see those cases that we've measured we see those cases that we've measured we see similar similar types of scaling laws a similar similar types of scaling laws a similar similar types of scaling laws a bit of a philosophical question but bit of a philosophical question but bit of a philosophical question but what's your intuition about why bigger what's your intuition about why bigger what's your intuition about why bigger is better in terms of network size and is better in terms of network size and is better in terms of network size and data size why does it lead to more data size why does it lead to more data size why does it lead to more intelligent models so in my previous intelligent models so in my previous intelligent models so in my previous career as a as a biophysicist so I did career as a as a biophysicist so I did career as a as a biophysicist so I did physics undergrad and then biophysics in physics undergrad and then biophysics in physics undergrad and then biophysics in in in in grad school so I think back to in in in grad school so I think back to in in in grad school so I think back to what I know as a physicist which is

what I know as a physicist which is what I know as a physicist which is actually much less than what some of my actually much less than what some of my actually much less than what some of my colleagues at anthropic have in terms of colleagues at anthropic have in terms of colleagues at anthropic have in terms of in terms of expertise in physics uh in terms of expertise in physics uh in terms of expertise in physics uh there's this there's this concept called there's this there's this concept called there's this there's this concept called the one over F noise and one overx the one over F noise and one overx the one over F noise and one overx distributions um where where often um uh distributions um where where often um uh distributions um where where often um uh you know just just like if you add up a you know just just like if you add up a you know just just like if you add up a bunch of natural processes you get bunch of natural processes you get bunch of natural processes you get gaussian if you add up a bunch of kind gaussian if you add up a bunch of kind gaussian if you add up a bunch of kind of differently distributed natural of differently distributed natural of differently distributed natural processes if you like if you like take a processes if you like if you like take a processes if you like if you like take a take a um probe and and hook it up to a take a um probe and and hook it up to a take a um probe and and hook it up to a resistor the distribution of the thermal resistor the distribution of the thermal resistor the distribution of the thermal noise in the resistor goes as one over noise in the resistor goes as one over noise in the resistor goes as one over the frequency um it's some kind of the frequency um it's some kind of the frequency um it's some kind of natural convergent distribution uh and natural convergent distribution uh and natural convergent distribution uh and and I I I I and and I think what it and I I I I and and I think what it and I I I I and and I think what it amounts to is that if you look at a lot amounts to is that if you look at a lot amounts to is that if you look at a lot of things that are that are produced by of things that are that are produced by of things that are that are produced by some natural process that has a lot of some natural process that has a lot of some natural process that has a lot of different scales right not a gaussian different scales right not a gaussian different scales right not a gaussian which is kind of narrowly distributed which is kind of narrowly distributed which is kind of narrowly distributed but you know if I look at kind of like but you know if I look at kind of like but you know if I look at kind of like large and small fluctuations that lead large and small fluctuations that lead large and small fluctuations that lead to lead to electrical noise um they have to lead to electrical noise um they have to lead to electrical noise um they have this decaying 1 overx distribution and this decaying 1 overx distribution and this decaying 1 overx distribution and so now I think of like patterns in the so now I think of like patterns in the so now I think of like patterns in the physical world right if I if or or in physical world right if I if or or in physical world right if I if or or in language if I think about the patterns language if I think about the patterns language if I think about the patterns in language there are some really simple in language there are some really simple in language there are some really simple patterns some words are much more common patterns some words are much more common patterns some words are much more common than others like the' then there's basic than others like the' then there's basic than others like the' then there's basic noun verb structure then there's the noun verb structure then there's the noun verb structure then there's the fact that you know you know nouns and fact that you know you know nouns and fact that you know you know nouns and verbs have to agree they have to verbs have to agree they have to verbs have to agree they have to coordinate and there's the higher level coordinate and there's the higher level coordinate and there's the higher level sentence structure then there's the sentence structure then there's the sentence structure then there's the Thematic structure of paragraphs and so Thematic structure of paragraphs and so Thematic structure of paragraphs and so the fact that there's this regressing the fact that there's this regressing the fact that there's this regressing structure you can imagine that as you

structure you can imagine that as you structure you can imagine that as you make the networks larger first they make the networks larger first they make the networks larger first they capture the really simple correlations capture the really simple correlations capture the really simple correlations the really simple patterns and there's the really simple patterns and there's the really simple patterns and there's this long taale of other patterns and if this long taale of other patterns and if this long taale of other patterns and if that long taale of other patterns is that long taale of other patterns is that long taale of other patterns is really smooth like it is with the one really smooth like it is with the one really smooth like it is with the one over F noise in you know physical over F noise in you know physical over F noise in you know physical processes like like like resistors then processes like like like resistors then processes like like like resistors then you could imagine as you make the you could imagine as you make the you could imagine as you make the network larger it's kind of capturing network larger it's kind of capturing network larger it's kind of capturing more and more of that distribution and more and more of that distribution and more and more of that distribution and so that smoothness gets reflected in how so that smoothness gets reflected in how so that smoothness gets reflected in how well the models are at predicting and well the models are at predicting and well the models are at predicting and how well they perform language is an how well they perform language is an how well they perform language is an evolved process right we've we've evolved process right we've we've evolved process right we've we've developed language we have common words developed language we have common words developed language we have common words and less common words we have common and less common words we have common and less common words we have common expressions and less common Expressions expressions and less common Expressions expressions and less common Expressions we have ideas cliches that are expressed we have ideas cliches that are expressed we have ideas cliches that are expressed frequently and we have novel ideas and frequently and we have novel ideas and frequently and we have novel ideas and that process has has developed has that process has has developed has that process has has developed has evolved with humans over millions of evolved with humans over millions of evolved with humans over millions of years and so the the the guess and this years and so the the the guess and this years and so the the the guess and this is pure speculation would be would be is pure speculation would be would be is pure speculation would be would be that there is there's some kind of that there is there's some kind of that there is there's some kind of longtail distribution of of of the longtail distribution of of of the longtail distribution of of of the distribution of these ideas so there's distribution of these ideas so there's distribution of these ideas so there's the long tail but also there's the the long tail but also there's the the long tail but also there's the height of the hierarchy of Concepts that height of the hierarchy of Concepts that height of the hierarchy of Concepts that you're building up so the bigger the you're building up so the bigger the you're building up so the bigger the network presumably you have a higher network presumably you have a higher network presumably you have a higher capacity to exactly if you have a small capacity to exactly if you have a small capacity to exactly if you have a small Network you only get the common stuff Network you only get the common stuff Network you only get the common stuff right if if I take a tiny neural network right if if I take a tiny neural network right if if I take a tiny neural network it's very good at understanding that you it's very good at understanding that you it's very good at understanding that you know a sentence has to have you know know a sentence has to have you know know a sentence has to have you know verb adjective noun right but it's it's verb adjective noun right but it's it's verb adjective noun right but it's it's terrible at deciding what those verb terrible at deciding what those verb terrible at deciding what those verb adjective and noun should be and whether adjective and noun should be and whether adjective and noun should be and whether they should make sense if I make it just they should make sense if I make it just they should make sense if I make it just a little bigger it gets good at that a little bigger it gets good at that a little bigger it gets good at that then suddenly it's good at the sentences

then suddenly it's good at the sentences then suddenly it's good at the sentences but it's not good at the paragraphs and but it's not good at the paragraphs and but it's not good at the paragraphs and so the these these rare and more complex so the these these rare and more complex so the these these rare and more complex patterns get picked up as I add as I add patterns get picked up as I add as I add patterns get picked up as I add as I add more capacity to the network well the more capacity to the network well the more capacity to the network well the natural question then is what's the natural question then is what's the natural question then is what's the ceiling of this like how complicated and ceiling of this like how complicated and ceiling of this like how complicated and complex is the real world how much of complex is the real world how much of complex is the real world how much of stuff is there to learn I don't think stuff is there to learn I don't think stuff is there to learn I don't think any of us knows the answer to that any of us knows the answer to that any of us knows the answer to that question um I my strong Instinct would question um I my strong Instinct would question um I my strong Instinct would be that there's no ceiling below level be that there's no ceiling below level be that there's no ceiling below level of humans right we humans are able to of humans right we humans are able to of humans right we humans are able to understand these various patterns and so understand these various patterns and so understand these various patterns and so that that makes me think that if we that that makes me think that if we that that makes me think that if we continue to you know scale up these continue to you know scale up these continue to you know scale up these these these models to kind of develop these these models to kind of develop these these models to kind of develop new methods for training them and new methods for training them and new methods for training them and scaling them up uh that will at least scaling them up uh that will at least scaling them up uh that will at least get to the level that we've gotten to get to the level that we've gotten to get to the level that we've gotten to with humans there's then a question of with humans there's then a question of with humans there's then a question of you know how much more is it possible to you know how much more is it possible to you know how much more is it possible to understand than humans do how much how understand than humans do how much how understand than humans do how much how much is it possible to be smarter and much is it possible to be smarter and much is it possible to be smarter and more perceptive than humans I I would more perceptive than humans I I would more perceptive than humans I I would guess the answer has has got to be guess the answer has has got to be guess the answer has has got to be domain dependent if I look at an area domain dependent if I look at an area domain dependent if I look at an area like biology and you know I wrote this like biology and you know I wrote this like biology and you know I wrote this essay Machines of Loving Grace it seems essay Machines of Loving Grace it seems essay Machines of Loving Grace it seems to me that humans are struggling to to me that humans are struggling to to me that humans are struggling to understand the complexity of biology understand the complexity of biology understand the complexity of biology right if you go to Stanford or to right if you go to Stanford or to right if you go to Stanford or to Harvard or to Berkeley you have whole Harvard or to Berkeley you have whole Harvard or to Berkeley you have whole Departments of you know folks trying to Departments of you know folks trying to Departments of you know folks trying to study you know like the immune system or study you know like the immune system or study you know like the immune system or metabolic pathways and and each person metabolic pathways and and each person metabolic pathways and and each person understands only a tiny bit part of it understands only a tiny bit part of it understands only a tiny bit part of it specializes and they're struggling to specializes and they're struggling to specializes and they're struggling to combine their knowledge with that of combine their knowledge with that of combine their knowledge with that of with that of other humans and so I have with that of other humans and so I have with that of other humans and so I have an instinct that there's there's a lot

an instinct that there's there's a lot an instinct that there's there's a lot of room at the top for AIS to get of room at the top for AIS to get of room at the top for AIS to get smarter if I think of something like smarter if I think of something like smarter if I think of something like materials in the in the physical world materials in the in the physical world materials in the in the physical world or you know um like addressing you know or you know um like addressing you know or you know um like addressing you know conflicts between humans or something conflicts between humans or something conflicts between humans or something like that I mean you know it it may be like that I mean you know it it may be like that I mean you know it it may be there's only some of these problems are there's only some of these problems are there's only some of these problems are not intractable but much harder and and not intractable but much harder and and not intractable but much harder and and it it may be that there's only there's it it may be that there's only there's it it may be that there's only there's only so well you can do with some of only so well you can do with some of only so well you can do with some of these things right just like with speech these things right just like with speech these things right just like with speech recognition there's only so clear I can recognition there's only so clear I can recognition there's only so clear I can hear your speech so I think in some hear your speech so I think in some hear your speech so I think in some areas there may be ceilings in in in you areas there may be ceilings in in in you areas there may be ceilings in in in you know that are very close to what humans know that are very close to what humans know that are very close to what humans have done in other areas those ceilings have done in other areas those ceilings have done in other areas those ceilings may be very far away and I think we'll may be very far away and I think we'll may be very far away and I think we'll only find out when we build these only find out when we build these only find out when we build these systems uh there's it's very hard to systems uh there's it's very hard to systems uh there's it's very hard to know in advance we can speculate but we know in advance we can speculate but we know in advance we can speculate but we can't be sure and in some domains the can't be sure and in some domains the can't be sure and in some domains the ceiling might have to do with human ceiling might have to do with human ceiling might have to do with human bureaucracies and things like this as bureaucracies and things like this as bureaucracies and things like this as you're right about yes so humans you're right about yes so humans you're right about yes so humans fundamentally have to be part of the fundamentally have to be part of the fundamentally have to be part of the loop that's the cause of the ceiling not loop that's the cause of the ceiling not loop that's the cause of the ceiling not maybe the limits of the intelligence maybe the limits of the intelligence maybe the limits of the intelligence yeah I think in many cases um you know yeah I think in many cases um you know yeah I think in many cases um you know in theory technology could change very in theory technology could change very in theory technology could change very fast for example all the things that we fast for example all the things that we fast for example all the things that we might invent with respect to biology um might invent with respect to biology um might invent with respect to biology um but remember there's there's a you know but remember there's there's a you know but remember there's there's a you know there's a clinical trial system that we there's a clinical trial system that we there's a clinical trial system that we have to go through to actually have to go through to actually have to go through to actually administer these things to humans I administer these things to humans I administer these things to humans I think that's a mixture of things that think that's a mixture of things that think that's a mixture of things that are unnecessary and bureaucratic and are unnecessary and bureaucratic and are unnecessary and bureaucratic and things that kind of protect the things that kind of protect the things that kind of protect the Integrity of society and the whole Integrity of society and the whole Integrity of society and the whole challenge is that it's hard to tell it's challenge is that it's hard to tell it's challenge is that it's hard to tell it's hard to tell what's going on uh it's hard to tell what's going on uh it's hard to tell what's going on uh it's hard to tell which is which right my my hard to tell which is which right my my hard to tell which is which right my my view is definitely I think in terms of view is definitely I think in terms of view is definitely I think in terms of drug development we my view is that

drug development we my view is that drug development we my view is that we're too slow and we're too we're too slow and we're too we're too slow and we're too conservative but certainly if you get conservative but certainly if you get conservative but certainly if you get these things wrong you know it's it's these things wrong you know it's it's these things wrong you know it's it's possible to to to risk people's lives by possible to to to risk people's lives by possible to to to risk people's lives by by being by being by being too Reckless by being by being by being too Reckless by being by being by being too Reckless and so at least at least some of these and so at least at least some of these and so at least at least some of these human institutions are in fact human institutions are in fact human institutions are in fact protecting people so it's it's all about protecting people so it's it's all about protecting people so it's it's all about finding the balance I strongly suspect finding the balance I strongly suspect finding the balance I strongly suspect that balance is kind of more on the side that balance is kind of more on the side that balance is kind of more on the side of pushing to make things happen faster of pushing to make things happen faster of pushing to make things happen faster but there is a balance if we do hit a but there is a balance if we do hit a but there is a balance if we do hit a limit if we do hit a Slowdown in the limit if we do hit a Slowdown in the limit if we do hit a Slowdown in the scaling laws what do you think would be scaling laws what do you think would be scaling laws what do you think would be the reason is it compute limited data the reason is it compute limited data the reason is it compute limited data limited uh is it something else idea limited uh is it something else idea limited uh is it something else idea limited so a few things now we're limited so a few things now we're limited so a few things now we're talking about hitting the limit before talking about hitting the limit before talking about hitting the limit before we get to the level of of humans and the we get to the level of of humans and the we get to the level of of humans and the skill of humans um so so I think one skill of humans um so so I think one skill of humans um so so I think one that's you know one that's popular today that's you know one that's popular today that's you know one that's popular today and I think you know could be a limit and I think you know could be a limit and I think you know could be a limit that we run into I like most of the that we run into I like most of the that we run into I like most of the limits I would bet against it but it's limits I would bet against it but it's limits I would bet against it but it's definitely possible is we simply run out definitely possible is we simply run out definitely possible is we simply run out of data there's only so much data on the of data there's only so much data on the of data there's only so much data on the internet and there's issues with the internet and there's issues with the internet and there's issues with the quality of the data right you can get quality of the data right you can get quality of the data right you can get hundreds of trillions of words on the hundreds of trillions of words on the hundreds of trillions of words on the internet but a lot of it is is internet but a lot of it is is internet but a lot of it is is repetitive or it's search engine you repetitive or it's search engine you repetitive or it's search engine you know search engine optimization driil or know search engine optimization driil or know search engine optimization driil or maybe in the future it'll even be text maybe in the future it'll even be text maybe in the future it'll even be text generated by AIS itself uh and and so I generated by AIS itself uh and and so I generated by AIS itself uh and and so I think there are limits to what to to think there are limits to what to to think there are limits to what to to what can be produced in this way that what can be produced in this way that what can be produced in this way that said we and I would guess other said we and I would guess other said we and I would guess other companies are working on ways to make companies are working on ways to make companies are working on ways to make data synthetic uh where you can you know data synthetic uh where you can you know data synthetic uh where you can you know you can use the model to generate more

you can use the model to generate more you can use the model to generate more data of the type that you have that you data of the type that you have that you data of the type that you have that you have already or even generate data from have already or even generate data from have already or even generate data from scratch if you think about uh what was scratch if you think about uh what was scratch if you think about uh what was done with uh deep mines Alpha go zero done with uh deep mines Alpha go zero done with uh deep mines Alpha go zero they managed to get a bot all the way they managed to get a bot all the way they managed to get a bot all the way from you know no ability to play Go from you know no ability to play Go from you know no ability to play Go whatsoever to above human level just by whatsoever to above human level just by whatsoever to above human level just by playing against itself there was no playing against itself there was no playing against itself there was no example data from humans required in the example data from humans required in the example data from humans required in the the alphao zero version of it the other the alphao zero version of it the other the alphao zero version of it the other direction of course is these reasoning direction of course is these reasoning direction of course is these reasoning models that do Chain of Thought and stop models that do Chain of Thought and stop models that do Chain of Thought and stop to think um and and reflect on their own to think um and and reflect on their own to think um and and reflect on their own thinking in a way that's another kind of thinking in a way that's another kind of thinking in a way that's another kind of synthetic data coupled with synthetic data coupled with synthetic data coupled with reinforcement learning so my my guess is reinforcement learning so my my guess is reinforcement learning so my my guess is with one of those methods we'll get with one of those methods we'll get with one of those methods we'll get around the data limitation or there may around the data limitation or there may around the data limitation or there may be other sources of data that are that be other sources of data that are that be other sources of data that are that are available um we could just observe are available um we could just observe are available um we could just observe that even if there's no problem with that even if there's no problem with that even if there's no problem with data as we start to scale models up they data as we start to scale models up they data as we start to scale models up they just stop getting better it's it seemed just stop getting better it's it seemed just stop getting better it's it seemed to be a a reliable observation that to be a a reliable observation that to be a a reliable observation that they've gotten better that could just they've gotten better that could just they've gotten better that could just stop at some point for a reason we don't stop at some point for a reason we don't stop at some point for a reason we don't understand um the answer could be that understand um the answer could be that understand um the answer could be that we need to uh you know we need to invent we need to uh you know we need to invent we need to uh you know we need to invent some new architecture um it's been there some new architecture um it's been there some new architecture um it's been there have been problems in the past with with have been problems in the past with with have been problems in the past with with say numerical stability of models where say numerical stability of models where say numerical stability of models where it looked like things were were leveling it looked like things were were leveling it looked like things were were leveling off but but actually you know know when off but but actually you know know when off but but actually you know know when we when we when we found the right we when we when we found the right we when we when we found the right Unblocker they didn't end up doing so so Unblocker they didn't end up doing so so Unblocker they didn't end up doing so so perhaps there's new some new perhaps there's new some new perhaps there's new some new optimization method or some new uh optimization method or some new uh optimization method or some new uh Technique we need to to unblock things Technique we need to to unblock things Technique we need to to unblock things I've seen no evidence of that so far but I've seen no evidence of that so far but I've seen no evidence of that so far but if things were to to slow down that if things were to to slow down that if things were to to slow down that perhaps could be one reason what about

perhaps could be one reason what about perhaps could be one reason what about the limits of compute meaning uh the the limits of compute meaning uh the the limits of compute meaning uh the expensive uh nature of building bigger expensive uh nature of building bigger expensive uh nature of building bigger and bigger data centers so right now I and bigger data centers so right now I and bigger data centers so right now I think uh you know most of the Frontier think uh you know most of the Frontier think uh you know most of the Frontier Model companies I would guess are are Model companies I would guess are are Model companies I would guess are are operating you know roughly you know $1 operating you know roughly you know $1 operating you know roughly you know $1 billion scale plus or minus a factor of billion scale plus or minus a factor of billion scale plus or minus a factor of three right those are the models that three right those are the models that three right those are the models that exist now or are being trained now uh I exist now or are being trained now uh I exist now or are being trained now uh I think next year we're going to go to a think next year we're going to go to a think next year we're going to go to a few billion and then uh 2026 we may go few billion and then uh 2026 we may go few billion and then uh 2026 we may go to uh uh you know above 10 10 10 billion to uh uh you know above 10 10 10 billion to uh uh you know above 10 10 10 billion and probably by 2027 their Ambitions to and probably by 2027 their Ambitions to and probably by 2027 their Ambitions to build hundred hundred billion dollar uh build hundred hundred billion dollar uh build hundred hundred billion dollar uh hundred billion dollar clusters and I hundred billion dollar clusters and I hundred billion dollar clusters and I think all of that actually will happen think all of that actually will happen think all of that actually will happen there's a lot of determination to build there's a lot of determination to build there's a lot of determination to build the compute to do it within this country the compute to do it within this country the compute to do it within this country uh and I would guess that it actually uh and I would guess that it actually uh and I would guess that it actually does happen now if we get to 100 billion does happen now if we get to 100 billion does happen now if we get to 100 billion that's still not enough compute that's that's still not enough compute that's that's still not enough compute that's still not enough scale then either we still not enough scale then either we still not enough scale then either we need even more scale or we need to need even more scale or we need to need even more scale or we need to develop some way of doing it more develop some way of doing it more develop some way of doing it more efficiently of Shifting The Curve um I efficiently of Shifting The Curve um I efficiently of Shifting The Curve um I think be between all of these one of the think be between all of these one of the think be between all of these one of the reasons I'm bullish about powerful AI reasons I'm bullish about powerful AI reasons I'm bullish about powerful AI happening so fast is just that if you happening so fast is just that if you happening so fast is just that if you extrapolate the next few points on the extrapolate the next few points on the extrapolate the next few points on the curve we're very quickly getting towards curve we're very quickly getting towards curve we're very quickly getting towards human level ability right some of the human level ability right some of the human level ability right some of the new models that that we developed some new models that that we developed some new models that that we developed some some reasoning models that have come some reasoning models that have come some reasoning models that have come from other companies they're starting to from other companies they're starting to from other companies they're starting to get to what I would call the PHD or get to what I would call the PHD or get to what I would call the PHD or professional level right if you look at professional level right if you look at professional level right if you look at their their coding ability um the latest their their coding ability um the latest their their coding ability um the latest model we released Sonet 3.5 the new or model we released Sonet 3.5 the new or model we released Sonet 3.5 the new or updated version it gets something like

updated version it gets something like updated version it gets something like 50% on sbench and sbench is an example 50% on sbench and sbench is an example 50% on sbench and sbench is an example of a bunch of professional real world of a bunch of professional real world of a bunch of professional real world software engineering tasks at the software engineering tasks at the software engineering tasks at the beginning of the year I think the beginning of the year I think the beginning of the year I think the state-of-the-art was three or 4% so in state-of-the-art was three or 4% so in state-of-the-art was three or 4% so in 10 months we've gone from 3% to 50% on 10 months we've gone from 3% to 50% on 10 months we've gone from 3% to 50% on this task and I think in another year this task and I think in another year this task and I think in another year we'll probably be at 90% I mean I don't we'll probably be at 90% I mean I don't we'll probably be at 90% I mean I don't know but might might even be might even know but might might even be might even know but might might even be might even be less than that uh we've seen similar be less than that uh we've seen similar be less than that uh we've seen similar things in graduate level math physics things in graduate level math physics things in graduate level math physics and biology from Models like open AI 01 and biology from Models like open AI 01 and biology from Models like open AI 01 uh so uh if we if we just continue to uh so uh if we if we just continue to uh so uh if we if we just continue to extrapolate this right in terms of skill extrapolate this right in terms of skill extrapolate this right in terms of skill skill that we have I think if we skill that we have I think if we skill that we have I think if we extrapolate the straight curve Within a extrapolate the straight curve Within a extrapolate the straight curve Within a few years we will get to these models few years we will get to these models few years we will get to these models being you know above the the highest being you know above the the highest being you know above the the highest professional level in terms of humans professional level in terms of humans professional level in terms of humans now will that curve continue you've now will that curve continue you've now will that curve continue you've pointed to and I've pointed to a lot of pointed to and I've pointed to a lot of pointed to and I've pointed to a lot of reasons why you know possible reasons reasons why you know possible reasons reasons why you know possible reasons why that might not happen but if the if why that might not happen but if the if why that might not happen but if the if the extrapolation curve continues that the extrapolation curve continues that the extrapolation curve continues that is the trajectory we're on so anthropic is the trajectory we're on so anthropic is the trajectory we're on so anthropic has several competitors it'd be has several competitors it'd be has several competitors it'd be interesting to get your sort of view of interesting to get your sort of view of interesting to get your sort of view of it all open aai Google xai meta what it all open aai Google xai meta what it all open aai Google xai meta what does it take to win in the broad sense does it take to win in the broad sense does it take to win in the broad sense of win in the space yeah so I want to of win in the space yeah so I want to of win in the space yeah so I want to separate out a couple things right so separate out a couple things right so separate out a couple things right so you know anthropics anthropic mission is you know anthropics anthropic mission is you know anthropics anthropic mission is to kind of try to make this all go well to kind of try to make this all go well to kind of try to make this all go well right and and you know we have a theory right and and you know we have a theory right and and you know we have a theory of change called race to the top right of change called race to the top right of change called race to the top right race to the top is about trying to push race to the top is about trying to push race to the top is about trying to push the other players to do the right thing

the other players to do the right thing the other players to do the right thing by setting an example it's not about by setting an example it's not about by setting an example it's not about being the good guy it's about setting being the good guy it's about setting being the good guy it's about setting things up so that all of us can be the things up so that all of us can be the things up so that all of us can be the good guy I'll give a few examples of good guy I'll give a few examples of good guy I'll give a few examples of this early in the history of anthropic this early in the history of anthropic this early in the history of anthropic one of our co-founders Chris Ola who I one of our co-founders Chris Ola who I one of our co-founders Chris Ola who I believe you're you're interviewing soon believe you're you're interviewing soon believe you're you're interviewing soon you know he's the co-founder of the you know he's the co-founder of the you know he's the co-founder of the field of mechanistic interpretability field of mechanistic interpretability field of mechanistic interpretability which is an attempt to understand what's which is an attempt to understand what's which is an attempt to understand what's going on inside AI models uh so we had going on inside AI models uh so we had going on inside AI models uh so we had him and one of our early teams focus on him and one of our early teams focus on him and one of our early teams focus on this area of interpretability which we this area of interpretability which we this area of interpretability which we think is good for making models safe and think is good for making models safe and think is good for making models safe and transparent for three or four years that transparent for three or four years that transparent for three or four years that had no commercial application whatsoever had no commercial application whatsoever had no commercial application whatsoever it still doesn't today we're doing some it still doesn't today we're doing some it still doesn't today we're doing some early betas with it and probably it will early betas with it and probably it will early betas with it and probably it will eventually but uh you know this is a eventually but uh you know this is a eventually but uh you know this is a very very long research bed in one in very very long research bed in one in very very long research bed in one in which we've we've built in public and which we've we've built in public and which we've we've built in public and shared our results publicly and and we shared our results publicly and and we shared our results publicly and and we did this because you know we think it's did this because you know we think it's did this because you know we think it's a way to make models safer an a way to make models safer an a way to make models safer an interesting thing is that as we've done interesting thing is that as we've done interesting thing is that as we've done this other companies have started doing this other companies have started doing this other companies have started doing it as well in some cases because they've it as well in some cases because they've it as well in some cases because they've been inspired by it in some cases been inspired by it in some cases been inspired by it in some cases because they're worried that uh you know because they're worried that uh you know because they're worried that uh you know if if other companies are doing this if if other companies are doing this if if other companies are doing this that look more responsible they want to that look more responsible they want to that look more responsible they want to look more responsible too no one wants look more responsible too no one wants look more responsible too no one wants to look like the irresponsible ible to look like the irresponsible ible to look like the irresponsible ible actor and and so they adopt this they actor and and so they adopt this they actor and and so they adopt this they adopt this as well when folks come to adopt this as well when folks come to adopt this as well when folks come to anthropic interpretability is often a anthropic interpretability is often a anthropic interpretability is often a draw and I tell them the other places draw and I tell them the other places draw and I tell them the other places you didn't go tell them why you came you didn't go tell them why you came you didn't go tell them why you came here um and and then you see soon that here um and and then you see soon that here um and and then you see soon that there that there's interpretability there that there's interpretability there that there's interpretability teams else elsewhere as well and in a teams else elsewhere as well and in a teams else elsewhere as well and in a way that takes away our competitive

way that takes away our competitive way that takes away our competitive Advantage because it's like oh they now Advantage because it's like oh they now Advantage because it's like oh they now others are doing it as well but it's others are doing it as well but it's others are doing it as well but it's good it's good for the broader system good it's good for the broader system good it's good for the broader system and so we have to invent some new thing and so we have to invent some new thing and so we have to invent some new thing that we're doing others aren't doing as that we're doing others aren't doing as that we're doing others aren't doing as well and the hope is to basically bid up well and the hope is to basically bid up well and the hope is to basically bid up bid up the importance of of of doing the bid up the importance of of of doing the bid up the importance of of of doing the right thing and it's not it's not about right thing and it's not it's not about right thing and it's not it's not about us in particular right it's not about us in particular right it's not about us in particular right it's not about having one particular good guy other having one particular good guy other having one particular good guy other companies can do this as well if they if companies can do this as well if they if companies can do this as well if they if they if they join the race to do this they if they join the race to do this they if they join the race to do this that's that's you know that's the best that's that's you know that's the best that's that's you know that's the best news ever right um uh it's it's just news ever right um uh it's it's just news ever right um uh it's it's just it's about kind of shaping the it's about kind of shaping the it's about kind of shaping the incentives to point upward instead of incentives to point upward instead of incentives to point upward instead of shaping the incentives to point to point shaping the incentives to point to point shaping the incentives to point to point downward and we should say this example downward and we should say this example downward and we should say this example the field of uh mechanistic the field of uh mechanistic the field of uh mechanistic interpretability is just a a rigorous interpretability is just a a rigorous interpretability is just a a rigorous non handwavy way of doing AI safety yes non handwavy way of doing AI safety yes non handwavy way of doing AI safety yes or it's tending that way trying to I or it's tending that way trying to I or it's tending that way trying to I mean I I think we're still early um in mean I I think we're still early um in mean I I think we're still early um in terms of our ability to see things but terms of our ability to see things but terms of our ability to see things but I've been surprised at how much we've I've been surprised at how much we've I've been surprised at how much we've been able to look inside these systems been able to look inside these systems been able to look inside these systems and understand what we see right unlike and understand what we see right unlike and understand what we see right unlike with the scaling laws where it feels with the scaling laws where it feels with the scaling laws where it feels like there's some you know law that's like there's some you know law that's like there's some you know law that's driving these models to perform better driving these models to perform better driving these models to perform better on on the inside the models aren't you on on the inside the models aren't you on on the inside the models aren't you know there's no reason why they should know there's no reason why they should know there's no reason why they should be designed for us to understand them be designed for us to understand them be designed for us to understand them right they're designed to operate right they're designed to operate right they're designed to operate they're designed to work just like the they're designed to work just like the they're designed to work just like the human brain or human biochemistry human brain or human biochemistry human brain or human biochemistry they're not designed for a human to open they're not designed for a human to open they're not designed for a human to open up the hatch look inside and understand up the hatch look inside and understand up the hatch look inside and understand them but we have found and you know you them but we have found and you know you them but we have found and you know you can talk in much more detail about this can talk in much more detail about this can talk in much more detail about this to Chris that when we open them up when to Chris that when we open them up when to Chris that when we open them up when we do look inside them we we find things we do look inside them we we find things we do look inside them we we find things that are surprisingly interesting and as

that are surprisingly interesting and as that are surprisingly interesting and as a side effect you also get to see the a side effect you also get to see the a side effect you also get to see the beauty of these models you get to beauty of these models you get to beauty of these models you get to explore the sort of uh the beautiful n explore the sort of uh the beautiful n explore the sort of uh the beautiful n nature of large neural networks through nature of large neural networks through nature of large neural networks through the me turb kind ofy I'm amazed at how the me turb kind ofy I'm amazed at how the me turb kind ofy I'm amazed at how clean it's been I I'm amazed at things clean it's been I I'm amazed at things clean it's been I I'm amazed at things like induction heads I'm amazed at like induction heads I'm amazed at like induction heads I'm amazed at things like uh you know that that we can things like uh you know that that we can things like uh you know that that we can you know use sparse autoencoders to find you know use sparse autoencoders to find you know use sparse autoencoders to find these directions within the networks uh these directions within the networks uh these directions within the networks uh and that the directions correspond to and that the directions correspond to and that the directions correspond to these very clear Concepts we these very clear Concepts we these very clear Concepts we demonstrated this a bit with the Golden demonstrated this a bit with the Golden demonstrated this a bit with the Golden Gate Bridge clad so this was an Gate Bridge clad so this was an Gate Bridge clad so this was an experiment where we found a direction experiment where we found a direction experiment where we found a direction inside one of the the neural network inside one of the the neural network inside one of the the neural network layers that corresponded to the Golden layers that corresponded to the Golden layers that corresponded to the Golden Gate Bridge and we just turned that way Gate Bridge and we just turned that way Gate Bridge and we just turned that way up and so we we released this model as a up and so we we released this model as a up and so we we released this model as a demo it was kind of half a joke uh for a demo it was kind of half a joke uh for a demo it was kind of half a joke uh for a couple days uh but it was it was couple days uh but it was it was couple days uh but it was it was illustrative of of the method we illustrative of of the method we illustrative of of the method we developed and uh you could you could developed and uh you could you could developed and uh you could you could take the Golden Gate you could take the take the Golden Gate you could take the take the Golden Gate you could take the model you could ask it about anything model you could ask it about anything model you could ask it about anything you know you know it would be like how you know you know it would be like how you know you know it would be like how you could say how was your day and you could say how was your day and you could say how was your day and anything you asked because this feature anything you asked because this feature anything you asked because this feature was activated would connect to the was activated would connect to the was activated would connect to the Golden Gate Bridge so it would say you Golden Gate Bridge so it would say you Golden Gate Bridge so it would say you know I'm I'm I'm feeling relaxed and know I'm I'm I'm feeling relaxed and know I'm I'm I'm feeling relaxed and expansive much like the the arches of expansive much like the the arches of expansive much like the the arches of the Golden Gate Bridge or you know it the Golden Gate Bridge or you know it the Golden Gate Bridge or you know it would masterfully change topic to the would masterfully change topic to the would masterfully change topic to the Golden Gate Bridge and it integrated Golden Gate Bridge and it integrated Golden Gate Bridge and it integrated there was also a sadness to it to to the there was also a sadness to it to to the there was also a sadness to it to to the focus ah had on the Golden Gate Bridge I focus ah had on the Golden Gate Bridge I focus ah had on the Golden Gate Bridge I think people quickly fell in love with think people quickly fell in love with think people quickly fell in love with it I think so people already miss it it I think so people already miss it it I think so people already miss it because it was taken down I think after because it was taken down I think after because it was taken down I think after a day somehow these interventions on the a day somehow these interventions on the a day somehow these interventions on the model um where where where where you

model um where where where where you model um where where where where you kind of adjust Its Behavior somehow kind of adjust Its Behavior somehow kind of adjust Its Behavior somehow emotionally made it seem more human than emotionally made it seem more human than emotionally made it seem more human than any other version of the model strong any other version of the model strong any other version of the model strong personality strong ID strong personality personality strong ID strong personality personality strong ID strong personality it has these kind of like obsessive it has these kind of like obsessive it has these kind of like obsessive interests you know we can all think of interests you know we can all think of interests you know we can all think of someone who's like obsessed with someone who's like obsessed with someone who's like obsessed with something so it does make it feel something so it does make it feel something so it does make it feel somehow a bit more human let's talk somehow a bit more human let's talk somehow a bit more human let's talk about the present let's talk about about the present let's talk about about the present let's talk about Claude so this year A lot has happened Claude so this year A lot has happened Claude so this year A lot has happened in March claw 3 Opa Sonet Hau were in March claw 3 Opa Sonet Hau were in March claw 3 Opa Sonet Hau were released then claw 35 Sonet in July with released then claw 35 Sonet in July with released then claw 35 Sonet in July with an updated version just now released and an updated version just now released and an updated version just now released and then also claw 35 hi coup was released then also claw 35 hi coup was released then also claw 35 hi coup was released okay can you explain the difference okay can you explain the difference okay can you explain the difference between Opus Sonet and Haiku and how we between Opus Sonet and Haiku and how we between Opus Sonet and Haiku and how we should think about the different should think about the different should think about the different versions yeah so let's go back to March versions yeah so let's go back to March versions yeah so let's go back to March when we first released uh these three when we first released uh these three when we first released uh these three models so you know our thinking was you models so you know our thinking was you models so you know our thinking was you different companies produce kind of different companies produce kind of different companies produce kind of large and small models better and worse large and small models better and worse large and small models better and worse models we felt that there was demand models we felt that there was demand models we felt that there was demand both for a really powerful model um you both for a really powerful model um you both for a really powerful model um you know and you that might be a little bit know and you that might be a little bit know and you that might be a little bit slower that you'd have to pay more for slower that you'd have to pay more for slower that you'd have to pay more for and also for fast cheap models that are and also for fast cheap models that are and also for fast cheap models that are as smart as they can be for how fast and as smart as they can be for how fast and as smart as they can be for how fast and cheap right whenever you want to do some cheap right whenever you want to do some cheap right whenever you want to do some kind of like you know difficult analysis kind of like you know difficult analysis kind of like you know difficult analysis like if I you know I want to write code like if I you know I want to write code like if I you know I want to write code for instance or you know I want to I for instance or you know I want to I for instance or you know I want to I want to brainstorm ideas or I want to do want to brainstorm ideas or I want to do want to brainstorm ideas or I want to do creative writing I want the really creative writing I want the really creative writing I want the really powerful model but then there's a lot of powerful model but then there's a lot of powerful model but then there's a lot of practical applications in a business practical applications in a business practical applications in a business sense where it's like I'm interacting

sense where it's like I'm interacting sense where it's like I'm interacting with a website I you know like I'm like with a website I you know like I'm like with a website I you know like I'm like doing my taxes or I'm you know talking doing my taxes or I'm you know talking doing my taxes or I'm you know talking to uh you know to like a legal adviser to uh you know to like a legal adviser to uh you know to like a legal adviser and I want to analyze a contract or you and I want to analyze a contract or you and I want to analyze a contract or you know we have plenty of companies that know we have plenty of companies that know we have plenty of companies that are just like you know you know I want are just like you know you know I want are just like you know you know I want to do autocomplete on my on my IDE or to do autocomplete on my on my IDE or to do autocomplete on my on my IDE or something uh and and for all of those something uh and and for all of those something uh and and for all of those things you want to act fast and you want things you want to act fast and you want things you want to act fast and you want to use the model very broadly so we to use the model very broadly so we to use the model very broadly so we wanted to serve that whole spectrum of wanted to serve that whole spectrum of wanted to serve that whole spectrum of needs um so we ended up with this uh you needs um so we ended up with this uh you needs um so we ended up with this uh you know this kind of poetry theme and so know this kind of poetry theme and so know this kind of poetry theme and so what's a really short poem it's a Haik what's a really short poem it's a Haik what's a really short poem it's a Haik cou and so Haiku is the small fast cheap cou and so Haiku is the small fast cheap cou and so Haiku is the small fast cheap model that is you know was at the time model that is you know was at the time model that is you know was at the time was released surprisingly surprisingly was released surprisingly surprisingly was released surprisingly surprisingly uh intelligent for how fast and cheap it uh intelligent for how fast and cheap it uh intelligent for how fast and cheap it was uh sonnet is a is a medium-sized was uh sonnet is a is a medium-sized was uh sonnet is a is a medium-sized poem right a couple paragraphs since o poem right a couple paragraphs since o poem right a couple paragraphs since o Sonet was the middle model it is smarter Sonet was the middle model it is smarter Sonet was the middle model it is smarter but also a little bit slower a little but also a little bit slower a little but also a little bit slower a little bit more expensive and and Opus like a bit more expensive and and Opus like a bit more expensive and and Opus like a magnum opus is a large work uh Opus was magnum opus is a large work uh Opus was magnum opus is a large work uh Opus was the the largest smartest model at the the the largest smartest model at the the the largest smartest model at the time um so that that was the original time um so that that was the original time um so that that was the original kind of thinking behind it um and our kind of thinking behind it um and our kind of thinking behind it um and our our thinking then was well each new our thinking then was well each new our thinking then was well each new generation of models should shift that generation of models should shift that generation of models should shift that tradeoff curve uh so when we release tradeoff curve uh so when we release tradeoff curve uh so when we release Sonet 3.5 it has the same roughly the Sonet 3.5 it has the same roughly the Sonet 3.5 it has the same roughly the same you know cost and speed as the same you know cost and speed as the same you know cost and speed as the Sonet 3 Model uh but uh it it increased Sonet 3 Model uh but uh it it increased Sonet 3 Model uh but uh it it increased its intelligence to the point where it its intelligence to the point where it its intelligence to the point where it was smarter than the original Opus 3 was smarter than the original Opus 3 was smarter than the original Opus 3 Model uh especially for code but but

Model uh especially for code but but Model uh especially for code but but also just in general and so now you know also just in general and so now you know also just in general and so now you know we've shown results for a Hau 3. 5 and I we've shown results for a Hau 3. 5 and I we've shown results for a Hau 3. 5 and I believe Hau 3.5 the smallest new model believe Hau 3.5 the smallest new model believe Hau 3.5 the smallest new model is about as good as Opus 3 the largest is about as good as Opus 3 the largest is about as good as Opus 3 the largest old model so basically the aim here is old model so basically the aim here is old model so basically the aim here is to shift the curve and then at some to shift the curve and then at some to shift the curve and then at some point there's going to be an opus 3.5 um point there's going to be an opus 3.5 um point there's going to be an opus 3.5 um now every new generation of models has now every new generation of models has now every new generation of models has its own thing they use new data their its own thing they use new data their its own thing they use new data their personality changes in ways that we kind personality changes in ways that we kind personality changes in ways that we kind of you know try to steer but are not of you know try to steer but are not of you know try to steer but are not fully able to steer and and so uh fully able to steer and and so uh fully able to steer and and so uh there's never quite that exact there's never quite that exact there's never quite that exact equivalence the only thing you're equivalence the only thing you're equivalence the only thing you're changing is intelligence um we always changing is intelligence um we always changing is intelligence um we always try and improve other things and some try and improve other things and some try and improve other things and some things change without us without us things change without us without us things change without us without us knowing or measuring so it's it's very knowing or measuring so it's it's very knowing or measuring so it's it's very much an inexact science in many ways the much an inexact science in many ways the much an inexact science in many ways the manner and personality of these models manner and personality of these models manner and personality of these models is more an art than it is a science so is more an art than it is a science so is more an art than it is a science so what is sort of the reason for uh the what is sort of the reason for uh the what is sort of the reason for uh the span of time between say Claude Opus 3 span of time between say Claude Opus 3 span of time between say Claude Opus 3 and 35 what is it what takes that time and 35 what is it what takes that time and 35 what is it what takes that time if you can speak to yeah so there's if you can speak to yeah so there's if you can speak to yeah so there's there's different there's different uh there's different there's different uh there's different there's different uh processes um uh there's pre-training processes um uh there's pre-training processes um uh there's pre-training which is you know just kind of the which is you know just kind of the which is you know just kind of the normal language model training and that normal language model training and that normal language model training and that takes a very long time um that uses you takes a very long time um that uses you takes a very long time um that uses you know these days you know tens you know know these days you know tens you know know these days you know tens you know tens of thousands sometimes many tens of tens of thousands sometimes many tens of tens of thousands sometimes many tens of thousands of uh gpus or tpus or tranium thousands of uh gpus or tpus or tranium thousands of uh gpus or tpus or tranium or you know what we use different or you know what we use different or you know what we use different platforms but you know accelerator chips

platforms but you know accelerator chips platforms but you know accelerator chips um often often training for months uh um often often training for months uh um often often training for months uh there's then a kind of posttraining there's then a kind of posttraining there's then a kind of posttraining phase where we do reinforcement learning phase where we do reinforcement learning phase where we do reinforcement learning from Human feedback as well as other from Human feedback as well as other from Human feedback as well as other kinds of reinforcement learning that kinds of reinforcement learning that kinds of reinforcement learning that that phase is getting uh larger and that phase is getting uh larger and that phase is getting uh larger and larger now and you know you know often larger now and you know you know often larger now and you know you know often that's less of an exact science it often that's less of an exact science it often that's less of an exact science it often takes effort to get it right um models takes effort to get it right um models takes effort to get it right um models are then tested with some of our early are then tested with some of our early are then tested with some of our early Partners to see how good they are and Partners to see how good they are and Partners to see how good they are and they're then tested both internally and they're then tested both internally and they're then tested both internally and externally for their safety particularly externally for their safety particularly externally for their safety particularly for catastrophic and autonomy r risks uh for catastrophic and autonomy r risks uh for catastrophic and autonomy r risks uh so uh we do internal testing according so uh we do internal testing according so uh we do internal testing according to our responsible scaling policy which to our responsible scaling policy which to our responsible scaling policy which I you know could talk more about that in I you know could talk more about that in I you know could talk more about that in detail and then we have an agreement detail and then we have an agreement detail and then we have an agreement with the US and the UK AI safety with the US and the UK AI safety with the US and the UK AI safety Institute as well as other third-party Institute as well as other third-party Institute as well as other third-party testers in specific domains to test the testers in specific domains to test the testers in specific domains to test the models for what are called cbrn risk models for what are called cbrn risk models for what are called cbrn risk chemical biological radiological and chemical biological radiological and chemical biological radiological and nuclear which are you know we don't nuclear which are you know we don't nuclear which are you know we don't think that models pose these risks think that models pose these risks think that models pose these risks seriously yet but but every new model we seriously yet but but every new model we seriously yet but but every new model we want to evaluate to see if we're want to evaluate to see if we're want to evaluate to see if we're starting to get close to some of these starting to get close to some of these starting to get close to some of these these these more dangerous um uh these these these more dangerous um uh these these these more dangerous um uh these more dangerous capabilities so those are more dangerous capabilities so those are more dangerous capabilities so those are the phases and then uh you know then the phases and then uh you know then the phases and then uh you know then then it just takes some time to get the then it just takes some time to get the then it just takes some time to get the model working in terms of inference and model working in terms of inference and model working in terms of inference and launching it in the API so there's just launching it in the API so there's just launching it in the API so there's just just a lot of steps to uh to actually to just a lot of steps to uh to actually to just a lot of steps to uh to actually to actually making a model work and of actually making a model work and of actually making a model work and of course you know we're always trying to course you know we're always trying to course you know we're always trying to make the processes as streamlined as

make the processes as streamlined as make the processes as streamlined as possible right we want our safety possible right we want our safety possible right we want our safety testing to be rigorous but we want it to testing to be rigorous but we want it to testing to be rigorous but we want it to be RoR ous and to be you know to be be RoR ous and to be you know to be be RoR ous and to be you know to be automatic to happen as fast as it can automatic to happen as fast as it can automatic to happen as fast as it can without compromising on rigor same with without compromising on rigor same with without compromising on rigor same with our pre-training process and our our pre-training process and our our pre-training process and our posttraining process so you know it's posttraining process so you know it's posttraining process so you know it's just like building anything else it's just like building anything else it's just like building anything else it's just like building airplanes you want to just like building airplanes you want to just like building airplanes you want to make them you know you want to make them make them you know you want to make them make them you know you want to make them safe but you want to make the process safe but you want to make the process safe but you want to make the process streamlined and I think the creative streamlined and I think the creative streamlined and I think the creative tension between those is is you know is tension between those is is you know is tension between those is is you know is an important thing and making the models an important thing and making the models an important thing and making the models work yeah uh rumor on the street I work yeah uh rumor on the street I work yeah uh rumor on the street I forget who was saying that uh anthropic forget who was saying that uh anthropic forget who was saying that uh anthropic is really good tooling so I uh probably is really good tooling so I uh probably is really good tooling so I uh probably a lot of the challenge here is on the a lot of the challenge here is on the a lot of the challenge here is on the software engineering side is to build software engineering side is to build software engineering side is to build the tooling to to have a like a the tooling to to have a like a the tooling to to have a like a efficient low friction interaction with efficient low friction interaction with efficient low friction interaction with the infrastructure you would be the infrastructure you would be the infrastructure you would be surprised how much of the challenges of surprised how much of the challenges of surprised how much of the challenges of uh you know building these models comes uh you know building these models comes uh you know building these models comes down to you know software engineering down to you know software engineering down to you know software engineering performance engineering you know you you performance engineering you know you you performance engineering you know you you know from the outside you might think oh know from the outside you might think oh know from the outside you might think oh man we had this Eureka breakthrough man we had this Eureka breakthrough man we had this Eureka breakthrough right you know this movie with the right you know this movie with the right you know this movie with the science we discovered it we figured it science we discovered it we figured it science we discovered it we figured it out but but but I think I think all out but but but I think I think all out but but but I think I think all things even even even you know things even even even you know things even even even you know incredible discoveries like they they incredible discoveries like they they incredible discoveries like they they they they they almost always come down they they they almost always come down they they they almost always come down to the details um and and often super to the details um and and often super to the details um and and often super super boring details I can't speak to super boring details I can't speak to super boring details I can't speak to whether we have better tooling than than whether we have better tooling than than whether we have better tooling than than other companies I mean you know I other companies I mean you know I other companies I mean you know I haven't been at those other companies at haven't been at those other companies at haven't been at those other companies at least at least not recently um but it's least at least not recently um but it's least at least not recently um but it's certainly something we give a lot of certainly something we give a lot of certainly something we give a lot of attention to I don't know if you can say attention to I don't know if you can say attention to I don't know if you can say but from three from CLA 3 to CLA 35 is but from three from CLA 3 to CLA 35 is but from three from CLA 3 to CLA 35 is there any extra pre-training going on or

there any extra pre-training going on or there any extra pre-training going on or is they mostly focus on the is they mostly focus on the is they mostly focus on the post-training there's been leaps in post-training there's been leaps in post-training there's been leaps in performance yeah I think I think at any performance yeah I think I think at any performance yeah I think I think at any given stage we're focused on improving given stage we're focused on improving given stage we're focused on improving everything at once um just just everything at once um just just everything at once um just just naturally like there are different teams naturally like there are different teams naturally like there are different teams each team makes progress in a particular each team makes progress in a particular each team makes progress in a particular area in in in making a particular you area in in in making a particular you area in in in making a particular you know their particular segment of the know their particular segment of the know their particular segment of the relay race better and it's just natural relay race better and it's just natural relay race better and it's just natural that when we make a new model we put we that when we make a new model we put we that when we make a new model we put we put all of these things in at once so put all of these things in at once so put all of these things in at once so the data you have like the preference the data you have like the preference the data you have like the preference data you get from rhf is that applicable data you get from rhf is that applicable data you get from rhf is that applicable is there ways to apply it to newer is there ways to apply it to newer is there ways to apply it to newer models as it get trained up yeah models as it get trained up yeah models as it get trained up yeah preference data from old models preference data from old models preference data from old models sometimes gets used for new models sometimes gets used for new models sometimes gets used for new models although of course uh it it performs although of course uh it it performs although of course uh it it performs somewhat better when it's you know somewhat better when it's you know somewhat better when it's you know trained on it's trained on the new trained on it's trained on the new trained on it's trained on the new models note that we have this you know models note that we have this you know models note that we have this you know constitutional AI method such that we constitutional AI method such that we constitutional AI method such that we don't only use preference data we kind don't only use preference data we kind don't only use preference data we kind of there's also a post-t trainining of there's also a post-t trainining of there's also a post-t trainining process where we train the model against process where we train the model against process where we train the model against itself and there's you know new types of itself and there's you know new types of itself and there's you know new types of post training the model against itself post training the model against itself post training the model against itself that are used every day so it's not just that are used every day so it's not just that are used every day so it's not just RF it's a bunch of other methods as well RF it's a bunch of other methods as well RF it's a bunch of other methods as well um post training I think you know it's um post training I think you know it's um post training I think you know it's becoming more and more sophisticated becoming more and more sophisticated becoming more and more sophisticated well what explains the big leap in well what explains the big leap in well what explains the big leap in performance for the new Sona 35 I mean performance for the new Sona 35 I mean performance for the new Sona 35 I mean at least in the programming side and at least in the programming side and at least in the programming side and maybe this is a good place to talk about maybe this is a good place to talk about maybe this is a good place to talk about benchmarks what does it mean to get benchmarks what does it mean to get benchmarks what does it mean to get better just the number went up but you better just the number went up but you better just the number went up but you know I I I program but I also love know I I I program but I also love know I I I program but I also love programming and I um claw 35 through programming and I um claw 35 through programming and I um claw 35 through cursor is what I use uh to assist me in cursor is what I use uh to assist me in cursor is what I use uh to assist me in programming and there was at least programming and there was at least programming and there was at least experientially anecdotally it's gotten

experientially anecdotally it's gotten experientially anecdotally it's gotten smarter at programming so what like what smarter at programming so what like what smarter at programming so what like what what does it take to get it uh to get it what does it take to get it uh to get it what does it take to get it uh to get it smarter we observe that as well by the smarter we observe that as well by the smarter we observe that as well by the way there were a couple uh very strong way there were a couple uh very strong way there were a couple uh very strong Engineers here at anthropic um who all Engineers here at anthropic um who all Engineers here at anthropic um who all previous code models both produced by us previous code models both produced by us previous code models both produced by us and produced by all the other companies and produced by all the other companies and produced by all the other companies hadn't really been useful to to hadn't hadn't really been useful to to hadn't hadn't really been useful to to hadn't really been useful to them you know they really been useful to them you know they really been useful to them you know they said you know maybe maybe this is useful said you know maybe maybe this is useful said you know maybe maybe this is useful to beginner it's not useful to me but to beginner it's not useful to me but to beginner it's not useful to me but Sonet 3.5 the original one for the first Sonet 3.5 the original one for the first Sonet 3.5 the original one for the first time they said oh my God this helped me time they said oh my God this helped me time they said oh my God this helped me with something that you know that it with something that you know that it with something that you know that it would have taken me hours to do this is would have taken me hours to do this is would have taken me hours to do this is the first model that has actually saved the first model that has actually saved the first model that has actually saved me time so again the water line is me time so again the water line is me time so again the water line is rising and and then I think you know the rising and and then I think you know the rising and and then I think you know the new Sonet has been has been even better new Sonet has been has been even better new Sonet has been has been even better in terms of what it what it takes I mean in terms of what it what it takes I mean in terms of what it what it takes I mean I'll just say it's been across the board I'll just say it's been across the board I'll just say it's been across the board it's in the pre-training it's in the it's in the pre-training it's in the it's in the pre-training it's in the posttraining it's in various evaluations posttraining it's in various evaluations posttraining it's in various evaluations that we do we've observed this as well that we do we've observed this as well that we do we've observed this as well and if we go into the details of the and if we go into the details of the and if we go into the details of the Benchmark so s bench is basically you Benchmark so s bench is basically you Benchmark so s bench is basically you know since since you know since since know since since you know since since know since since you know since since you're a programmer you know you'll be you're a programmer you know you'll be you're a programmer you know you'll be familiar with like PLL requests and you familiar with like PLL requests and you familiar with like PLL requests and you know uh just just PLL requests are like know uh just just PLL requests are like know uh just just PLL requests are like you know the like a sort of a sort of you know the like a sort of a sort of you know the like a sort of a sort of atomic unit of work you know you could atomic unit of work you know you could atomic unit of work you know you could say I'm you know I'm implementing one say I'm you know I'm implementing one say I'm you know I'm implementing one I'm implementing one thing um uh and and I'm implementing one thing um uh and and I'm implementing one thing um uh and and so sbench actually gives you kind of a so sbench actually gives you kind of a so sbench actually gives you kind of a real world situation where the codebase real world situation where the codebase real world situation where the codebase is in a current state and I'm trying to is in a current state and I'm trying to is in a current state and I'm trying to implement something that's you know implement something that's you know implement something that's you know that's described in described in that's described in described in that's described in described in language we have internal benchmarks language we have internal benchmarks language we have internal benchmarks where we where we measure the same thing where we where we measure the same thing where we where we measure the same thing and you say just give the model free and you say just give the model free and you say just give the model free reign to like you know do anything run

reign to like you know do anything run reign to like you know do anything run run run anything edit anything um how run run anything edit anything um how run run anything edit anything um how how well is it able to complete these how well is it able to complete these how well is it able to complete these tasks and it's that Benchmark that's tasks and it's that Benchmark that's tasks and it's that Benchmark that's gone from it can do it 3% of the time to gone from it can do it 3% of the time to gone from it can do it 3% of the time to it can do it about 50% of the time um so it can do it about 50% of the time um so it can do it about 50% of the time um so I actually do believe that if we get you I actually do believe that if we get you I actually do believe that if we get you can gain benchmarks but I think if we can gain benchmarks but I think if we can gain benchmarks but I think if we get to 100% on that Benchmark in a way get to 100% on that Benchmark in a way get to 100% on that Benchmark in a way that isn't kind of like overtrained or that isn't kind of like overtrained or that isn't kind of like overtrained or or or game for that particular Benchmark or or game for that particular Benchmark or or game for that particular Benchmark probably represents a real and serious probably represents a real and serious probably represents a real and serious increase in kind of increase in kind of increase in kind of in kind of programming programming in kind of programming programming in kind of programming programming ability and and I would suspect that if ability and and I would suspect that if ability and and I would suspect that if we can get to you know 90 90 95% that we can get to you know 90 90 95% that we can get to you know 90 90 95% that that that that you know it will it will that that that you know it will it will that that that you know it will it will represent ability to autonomously do a represent ability to autonomously do a represent ability to autonomously do a significant fraction of software significant fraction of software significant fraction of software engineering engineering engineering tasks well ridiculous timeline question tasks well ridiculous timeline question tasks well ridiculous timeline question uh when is clad Opus uh 3.5 coming up uh uh when is clad Opus uh 3.5 coming up uh uh when is clad Opus uh 3.5 coming up uh not giving you an exact date uh but you not giving you an exact date uh but you not giving you an exact date uh but you know there there uh you know as far as know there there uh you know as far as know there there uh you know as far as we know the plan is still to have a we know the plan is still to have a we know the plan is still to have a Claude 3.5 opus are we gonna get it Claude 3.5 opus are we gonna get it Claude 3.5 opus are we gonna get it before GTA 6 or no like Duke Nukem before GTA 6 or no like Duke Nukem before GTA 6 or no like Duke Nukem Forever was that game that there was Forever was that game that there was Forever was that game that there was some game that was delayed 15 years was some game that was delayed 15 years was some game that was delayed 15 years was that Duke Nukem Forever yeah and I think that Duke Nukem Forever yeah and I think that Duke Nukem Forever yeah and I think GTA is now just releasing trailers it GTA is now just releasing trailers it GTA is now just releasing trailers it you know it's only been three months you know it's only been three months you know it's only been three months since we released the first son it yeah since we released the first son it yeah since we released the first son it yeah it's Inc the incredible pace of relas it it's Inc the incredible pace of relas it it's Inc the incredible pace of relas it just it just tells you about the pace just it just tells you about the pace just it just tells you about the pace the expectations for when things are the expectations for when things are the expectations for when things are going to come out so uh what about going to come out so uh what about going to come out so uh what about 40 so how do you think about sort of as 40 so how do you think about sort of as 40 so how do you think about sort of as these models get bigger and bigger about these models get bigger and bigger about these models get bigger and bigger about versioning and also just versioning in versioning and also just versioning in versioning and also just versioning in general why Sonet 35 updated with the

general why Sonet 35 updated with the general why Sonet 35 updated with the date why not Sonet date why not Sonet date why not Sonet 3.6 actually naming is actually an 3.6 actually naming is actually an 3.6 actually naming is actually an interesting challenge here right because interesting challenge here right because interesting challenge here right because I think a year ago most of the model was I think a year ago most of the model was I think a year ago most of the model was pre-training and so you could start from pre-training and so you could start from pre-training and so you could start from the beginning and just say okay we're the beginning and just say okay we're the beginning and just say okay we're going to have models of different sizes going to have models of different sizes going to have models of different sizes we're going to train them all together we're going to train them all together we're going to train them all together and you know we'll have a a family of and you know we'll have a a family of and you know we'll have a a family of naming schemes and then we'll put some naming schemes and then we'll put some naming schemes and then we'll put some new magic into them and then you know new magic into them and then you know new magic into them and then you know we'll have the next the next Generation we'll have the next the next Generation we'll have the next the next Generation Um the trouble starts are already when Um the trouble starts are already when Um the trouble starts are already when some of them take a lot longer than some of them take a lot longer than some of them take a lot longer than others to train right that already others to train right that already others to train right that already messes up your time time a little bit messes up your time time a little bit messes up your time time a little bit but as you make big improvements in as but as you make big improvements in as but as you make big improvements in as you make big improvements in you make big improvements in you make big improvements in pre-training uh then you suddenly notice pre-training uh then you suddenly notice pre-training uh then you suddenly notice oh I can make better pre-train model and oh I can make better pre-train model and oh I can make better pre-train model and that doesn't take very long to do and that doesn't take very long to do and that doesn't take very long to do and but you know clearly it has the same you but you know clearly it has the same you but you know clearly it has the same you know size and shape of previous models know size and shape of previous models know size and shape of previous models uh uh so I think those two together as uh uh so I think those two together as uh uh so I think those two together as well as the timing timing issues any well as the timing timing issues any well as the timing timing issues any kind of scheme you come up with uh you kind of scheme you come up with uh you kind of scheme you come up with uh you know the reality tends to kind of know the reality tends to kind of know the reality tends to kind of frustrate that scheme right T tends to frustrate that scheme right T tends to frustrate that scheme right T tends to kind of break out of the break out of kind of break out of the break out of kind of break out of the break out of the scheme it's not like software where the scheme it's not like software where the scheme it's not like software where you can say oh this is like you know 3.7 you can say oh this is like you know 3.7 you can say oh this is like you know 3.7 this is 3.8 no you have models with this is 3.8 no you have models with this is 3.8 no you have models with different different tradeoffs you can different different tradeoffs you can different different tradeoffs you can change some things in your models you change some things in your models you change some things in your models you can train you can change other things can train you can change other things can train you can change other things some are faster and slower at inference some are faster and slower at inference some are faster and slower at inference some have to be more expensive some have some have to be more expensive some have some have to be more expensive some have to be less expensive and so I think all to be less expensive and so I think all to be less expensive and so I think all the companies have struggled with this the companies have struggled with this the companies have struggled with this um I think we did very you know I think um I think we did very you know I think um I think we did very you know I think think we were in a good good position in think we were in a good good position in think we were in a good good position in terms of naming when we had Haiku Sonet terms of naming when we had Haiku Sonet terms of naming when we had Haiku Sonet and we're trying to maintain it but it's

and we're trying to maintain it but it's and we're trying to maintain it but it's not it's not it's not perfect um so not it's not it's not perfect um so not it's not it's not perfect um so we'll we'll we'll try and get back to we'll we'll we'll try and get back to we'll we'll we'll try and get back to the Simplicity but it it um uh just the the Simplicity but it it um uh just the the Simplicity but it it um uh just the the the nature of the field I feel like the the nature of the field I feel like the the nature of the field I feel like no one's figured out naming it's somehow no one's figured out naming it's somehow no one's figured out naming it's somehow a different Paradigm from like normal a different Paradigm from like normal a different Paradigm from like normal software and and and so we we just none software and and and so we we just none software and and and so we we just none of the companies have been perfect at it of the companies have been perfect at it of the companies have been perfect at it um it's something we struggle with um it's something we struggle with um it's something we struggle with surprisingly much relative to you know surprisingly much relative to you know surprisingly much relative to you know how relative to how trivial it is to you how relative to how trivial it is to you how relative to how trivial it is to you know for the the the the grand science know for the the the the grand science know for the the the the grand science of training the models so from the user of training the models so from the user of training the models so from the user side the user experience of the updated side the user experience of the updated side the user experience of the updated Sonet 35 is just different than the Sonet 35 is just different than the Sonet 35 is just different than the previous uh June 2024 Sonet 35 it would previous uh June 2024 Sonet 35 it would previous uh June 2024 Sonet 35 it would be nice to come up with some kind of be nice to come up with some kind of be nice to come up with some kind of labeling that embodies that because labeling that embodies that because labeling that embodies that because people talk about son 35 but now there's people talk about son 35 but now there's people talk about son 35 but now there's a different one and so how do you refer a different one and so how do you refer a different one and so how do you refer to the previous one and the new one and to the previous one and the new one and to the previous one and the new one and it it uh when there's a distinct it it uh when there's a distinct it it uh when there's a distinct Improvement it just makes conversation Improvement it just makes conversation Improvement it just makes conversation about it uh just challenging yeah yeah I about it uh just challenging yeah yeah I about it uh just challenging yeah yeah I I definitely think this question of I definitely think this question of I definitely think this question of there are lots of properties of the there are lots of properties of the there are lots of properties of the models that are not reflected in the models that are not reflected in the models that are not reflected in the benchmarks um I I think I think that's benchmarks um I I think I think that's benchmarks um I I think I think that's that's definitely the case and everyone that's definitely the case and everyone that's definitely the case and everyone agrees and not all of them are agrees and not all of them are agrees and not all of them are capabilities some of them are you know capabilities some of them are you know capabilities some of them are you know models can be polite or brusk they can models can be polite or brusk they can models can be polite or brusk they can be uh you know uh very reactive or they be uh you know uh very reactive or they be uh you know uh very reactive or they can ask you questions um they can have can ask you questions um they can have can ask you questions um they can have what what feels like a warm personality what what feels like a warm personality what what feels like a warm personality or a cold personality they can be boring

or a cold personality they can be boring or a cold personality they can be boring or they can be very distinctive like or they can be very distinctive like or they can be very distinctive like Golden Gate Claude was um and we have a Golden Gate Claude was um and we have a Golden Gate Claude was um and we have a whole you know we have a whole team kind whole you know we have a whole team kind whole you know we have a whole team kind of focused on I think we call it Claude of focused on I think we call it Claude of focused on I think we call it Claude character uh Amanda leads that team and character uh Amanda leads that team and character uh Amanda leads that team and we'll we'll talk to you about that but we'll we'll talk to you about that but we'll we'll talk to you about that but it's still a very inexact science um and it's still a very inexact science um and it's still a very inexact science um and and often we find that models have and often we find that models have and often we find that models have properties that we're not aware of the properties that we're not aware of the properties that we're not aware of the the fact of the matter is that you can the fact of the matter is that you can the fact of the matter is that you can you know talk to a model 10,000 times you know talk to a model 10,000 times you know talk to a model 10,000 times and there are some behaviors you might and there are some behaviors you might and there are some behaviors you might not see uh just like just like with a not see uh just like just like with a not see uh just like just like with a human right I can know someone for a few human right I can know someone for a few human right I can know someone for a few months and you know not know that they months and you know not know that they months and you know not know that they have a certain skill or not know there's have a certain skill or not know there's have a certain skill or not know there's a certain side to them and so I think I a certain side to them and so I think I a certain side to them and so I think I think we just have to get used to this think we just have to get used to this think we just have to get used to this idea and we're always looking for better idea and we're always looking for better idea and we're always looking for better ways of testing our models to to ways of testing our models to to ways of testing our models to to demonstrate these capabilities and and demonstrate these capabilities and and demonstrate these capabilities and and and also to decide which are which are and also to decide which are which are and also to decide which are which are the which are the personality properties the which are the personality properties the which are the personality properties we want models to have have and which we we want models to have have and which we we want models to have have and which we don't want to have that itself the don't want to have that itself the don't want to have that itself the normative question is also super normative question is also super normative question is also super interesting I got to ask you a question interesting I got to ask you a question interesting I got to ask you a question from Reddit from Reddit oh from Reddit from Reddit oh from Reddit from Reddit oh boy you know there there's just this boy you know there there's just this boy you know there there's just this fascinating to me at least it's a fascinating to me at least it's a fascinating to me at least it's a psychological social psychological social psychological social phenomenon where people report that phenomenon where people report that phenomenon where people report that Claude has gotten Dumber for them over Claude has gotten Dumber for them over Claude has gotten Dumber for them over time and so uh the question is does the time and so uh the question is does the time and so uh the question is does the user complaint about the dumbing down of user complaint about the dumbing down of user complaint about the dumbing down of claw 35 Sonic hold any water so are claw 35 Sonic hold any water so are claw 35 Sonic hold any water so are these anecdota reports a kind of social these anecdota reports a kind of social these anecdota reports a kind of social phenomena or did Claude is there any phenomena or did Claude is there any phenomena or did Claude is there any cases where Claude would get Dumber so cases where Claude would get Dumber so cases where Claude would get Dumber so uh this actually doesn't apply this this uh this actually doesn't apply this this uh this actually doesn't apply this this isn't just about Claude I I believe this

isn't just about Claude I I believe this isn't just about Claude I I believe this I believe I've seen these complaints for I believe I've seen these complaints for I believe I've seen these complaints for every Foundation model produced by a every Foundation model produced by a every Foundation model produced by a major company um people said this about major company um people said this about major company um people said this about gp4 they said it about gp4 turbo um so gp4 they said it about gp4 turbo um so gp4 they said it about gp4 turbo um so so so a couple things um one the actual so so a couple things um one the actual so so a couple things um one the actual weights of the model right the actual weights of the model right the actual weights of the model right the actual brain of the model that does not change brain of the model that does not change brain of the model that does not change unless we introduce a new model um there unless we introduce a new model um there unless we introduce a new model um there there just a number of reasons why it there just a number of reasons why it there just a number of reasons why it would not make sense practically to be would not make sense practically to be would not make sense practically to be randomly substituting in substituting in randomly substituting in substituting in randomly substituting in substituting in new versions of the model it's difficult new versions of the model it's difficult new versions of the model it's difficult from an inference perspective and it's from an inference perspective and it's from an inference perspective and it's actually hard to control all the actually hard to control all the actually hard to control all the consequences of changing the way to the consequences of changing the way to the consequences of changing the way to the model let's say you wanted to fine-tune model let's say you wanted to fine-tune model let's say you wanted to fine-tune the model to be like I don't know to the model to be like I don't know to the model to be like I don't know to like to say certainly less which you like to say certainly less which you like to say certainly less which you know an old version of Sonet used to do know an old version of Sonet used to do know an old version of Sonet used to do um you actually end up changing a 100 um you actually end up changing a 100 um you actually end up changing a 100 things as well so we have a whole things as well so we have a whole things as well so we have a whole process for it and we have a whole process for it and we have a whole process for it and we have a whole process for modifying the model we do a process for modifying the model we do a process for modifying the model we do a bunch of testing on it we do a bunch of bunch of testing on it we do a bunch of bunch of testing on it we do a bunch of um like we do a bunch of user testing um like we do a bunch of user testing um like we do a bunch of user testing and early customers so it we both have and early customers so it we both have and early customers so it we both have never changed the weights of the model never changed the weights of the model never changed the weights of the model without without telling anyone and it it without without telling anyone and it it without without telling anyone and it it it wouldn't certainly in the current it wouldn't certainly in the current it wouldn't certainly in the current setup it would not make sense to do that setup it would not make sense to do that setup it would not make sense to do that now there are a couple things that we do now there are a couple things that we do now there are a couple things that we do occasionally do um one is sometimes we occasionally do um one is sometimes we occasionally do um one is sometimes we run AB tests um but those are typically run AB tests um but those are typically run AB tests um but those are typically very close to when a model is being is very close to when a model is being is very close to when a model is being is being uh released and for a very small being uh released and for a very small being uh released and for a very small fraction of time um so uh you know like fraction of time um so uh you know like fraction of time um so uh you know like the you know the the day before the new the you know the the day before the new the you know the the day before the new Sonet 3.5 I I agree we should have Sonet 3.5 I I agree we should have Sonet 3.5 I I agree we should have should have had a better name it's should have had a better name it's should have had a better name it's clunky to refer to it um there were some

clunky to refer to it um there were some clunky to refer to it um there were some comments from people that like it's got comments from people that like it's got comments from people that like it's got It's got it's gotten a lot better and It's got it's gotten a lot better and It's got it's gotten a lot better and that's because you know a fraction were that's because you know a fraction were that's because you know a fraction were exposed to to an AB test for for those exposed to to an AB test for for those exposed to to an AB test for for those one or for those one or two days um the one or for those one or two days um the one or for those one or two days um the other is that occasionally the system other is that occasionally the system other is that occasionally the system prompt will change um on the system prompt will change um on the system prompt will change um on the system prompt can have some effects although prompt can have some effects although prompt can have some effects although it's un it it it's unlikely to dumb down it's un it it it's unlikely to dumb down it's un it it it's unlikely to dumb down models it's unlikely to make them Dumber models it's unlikely to make them Dumber models it's unlikely to make them Dumber um and and and and we've seen that while um and and and and we've seen that while um and and and and we've seen that while these two things which I'm listing to be these two things which I'm listing to be these two things which I'm listing to be very complete um happen relatively very complete um happen relatively very complete um happen relatively happen quite infrequently um the happen quite infrequently um the happen quite infrequently um the complaints about to for us and for other complaints about to for us and for other complaints about to for us and for other model companies about the model changed model companies about the model changed model companies about the model changed the model isn't good at this the model the model isn't good at this the model the model isn't good at this the model got more censored the model was dumb got more censored the model was dumb got more censored the model was dumb down those complaints are constant and down those complaints are constant and down those complaints are constant and so I don't want to say like people are so I don't want to say like people are so I don't want to say like people are imagining it or anything but like the imagining it or anything but like the imagining it or anything but like the models are for the most part not models are for the most part not models are for the most part not changing um if I were to offer a theory changing um if I were to offer a theory changing um if I were to offer a theory um I I think it actually relates to one um I I think it actually relates to one um I I think it actually relates to one of the things I said before which is of the things I said before which is of the things I said before which is that models have many are very complex that models have many are very complex that models have many are very complex and have many aspects to them and so and have many aspects to them and so and have many aspects to them and so often you know if I if I if if I ask a often you know if I if I if if I ask a often you know if I if I if if I ask a model a question you know if I'm like if model a question you know if I'm like if model a question you know if I'm like if I'm like do task X versus can you do I'm like do task X versus can you do I'm like do task X versus can you do task XX the model might respond in task XX the model might respond in task XX the model might respond in different ways uh and and so there are different ways uh and and so there are different ways uh and and so there are all kinds of subtle things that you can all kinds of subtle things that you can all kinds of subtle things that you can change about the way you interact with change about the way you interact with change about the way you interact with the model that can give you very the model that can give you very the model that can give you very different results um to be clear this different results um to be clear this different results um to be clear this this itself is like a failing by by us this itself is like a failing by by us this itself is like a failing by by us and by the other model providers that and by the other model providers that and by the other model providers that that the models are are just just often that the models are are just just often that the models are are just just often sensitive to like small small changes in

sensitive to like small small changes in sensitive to like small small changes in wording it's yet another way in which wording it's yet another way in which wording it's yet another way in which the science of how these models work is the science of how these models work is the science of how these models work is very poorly developed uh and and so you very poorly developed uh and and so you very poorly developed uh and and so you know if I go to sleep one night and I know if I go to sleep one night and I know if I go to sleep one night and I was like talking to the model in a was like talking to the model in a was like talking to the model in a certain way and I like slightly Chang certain way and I like slightly Chang certain way and I like slightly Chang the phrasing of how I talk to the model the phrasing of how I talk to the model the phrasing of how I talk to the model you know I could I could get different you know I could I could get different you know I could I could get different results so that's that's one possible results so that's that's one possible results so that's that's one possible way the other thing is man it's just way the other thing is man it's just way the other thing is man it's just hard to quantify this stuff uh it's hard hard to quantify this stuff uh it's hard hard to quantify this stuff uh it's hard to quantify this stuff I think people to quantify this stuff I think people to quantify this stuff I think people are very excited by new models when they are very excited by new models when they are very excited by new models when they come out and then as time goes on they come out and then as time goes on they come out and then as time goes on they they become very aware of the they they become very aware of the they they become very aware of the they become very aware of the limitations so become very aware of the limitations so become very aware of the limitations so that may be another effect but that's that may be another effect but that's that may be another effect but that's that's all a very long- rended way of that's all a very long- rended way of that's all a very long- rended way of saying for the most part with some saying for the most part with some saying for the most part with some fairly narrow exceptions the models are fairly narrow exceptions the models are fairly narrow exceptions the models are not changing I think there is a not changing I think there is a not changing I think there is a psychological effect you just start psychological effect you just start psychological effect you just start getting used to it the Baseline ra like getting used to it the Baseline ra like getting used to it the Baseline ra like when people have first gotten Wi-Fi on when people have first gotten Wi-Fi on when people have first gotten Wi-Fi on airplanes it's like amazing magic and airplanes it's like amazing magic and airplanes it's like amazing magic and then now like I can't get this thing to then now like I can't get this thing to then now like I can't get this thing to work this is such a piece of crap work this is such a piece of crap work this is such a piece of crap exactly so it's easy to have the exactly so it's easy to have the exactly so it's easy to have the conspiracy theory of they're making conspiracy theory of they're making conspiracy theory of they're making Wi-Fi slower and slower this is probably Wi-Fi slower and slower this is probably Wi-Fi slower and slower this is probably something I'll talk to Amanda much more something I'll talk to Amanda much more something I'll talk to Amanda much more about but U another Reddit question uh about but U another Reddit question uh about but U another Reddit question uh when will Claud stop trying to be my uh when will Claud stop trying to be my uh when will Claud stop trying to be my uh panical grandmother imposing its moral panical grandmother imposing its moral panical grandmother imposing its moral World viw on me as a paying customer and World viw on me as a paying customer and World viw on me as a paying customer and also what does it that ology behind also what does it that ology behind also what does it that ology behind making Claude overly apologetic so this making Claude overly apologetic so this making Claude overly apologetic so this kind of reports about The Experience a kind of reports about The Experience a kind of reports about The Experience a different angle on the frustration it different angle on the frustration it different angle on the frustration it has to do with the character yeah so a has to do with the character yeah so a has to do with the character yeah so a couple points on this first one is um couple points on this first one is um couple points on this first one is um like things that people say on Reddit like things that people say on Reddit like things that people say on Reddit and Twitter or X or whatever it is um

and Twitter or X or whatever it is um and Twitter or X or whatever it is um there's actually a huge distribution there's actually a huge distribution there's actually a huge distribution shift between like the stuff that people shift between like the stuff that people shift between like the stuff that people complain loudly about on social media complain loudly about on social media complain loudly about on social media and what actually kind of like you know and what actually kind of like you know and what actually kind of like you know statistically users care about and that statistically users care about and that statistically users care about and that drives people to use the models like drives people to use the models like drives people to use the models like people are frustrated with you know people are frustrated with you know people are frustrated with you know things like you know the model not things like you know the model not things like you know the model not writing out all the code or the model uh writing out all the code or the model uh writing out all the code or the model uh you know just just not being as good at you know just just not being as good at you know just just not being as good at code as it could be even though it's the code as it could be even though it's the code as it could be even though it's the best model in the world on code um I I best model in the world on code um I I best model in the world on code um I I think the majority of thing of things think the majority of thing of things think the majority of thing of things are about that um uh but uh certainly a are about that um uh but uh certainly a are about that um uh but uh certainly a a a kind of vocal minority are uh you a a kind of vocal minority are uh you a a kind of vocal minority are uh you know kind kind of kind of rais these know kind kind of kind of rais these know kind kind of kind of rais these concerns right are frustrated by the concerns right are frustrated by the concerns right are frustrated by the model refusing things that it shouldn't model refusing things that it shouldn't model refusing things that it shouldn't refuse or like apologizing too much or refuse or like apologizing too much or refuse or like apologizing too much or just just having these kind of like just just having these kind of like just just having these kind of like annoying verbal ticks um the second annoying verbal ticks um the second annoying verbal ticks um the second caveat and I just want to say this like caveat and I just want to say this like caveat and I just want to say this like super clearly because I think it's like super clearly because I think it's like super clearly because I think it's like some people don't know it others like some people don't know it others like some people don't know it others like kind of know it but forget it like it is kind of know it but forget it like it is kind of know it but forget it like it is very difficult to control across the very difficult to control across the very difficult to control across the board how the models behave you cannot board how the models behave you cannot board how the models behave you cannot just reach in there and say oh I want just reach in there and say oh I want just reach in there and say oh I want the model to like apologize less like the model to like apologize less like the model to like apologize less like you can do that you can include trading you can do that you can include trading you can do that you can include trading data that says like oh the models should data that says like oh the models should data that says like oh the models should like apologize less but then in some like apologize less but then in some like apologize less but then in some other situation they end up being like other situation they end up being like other situation they end up being like super rude or like overconfident in a super rude or like overconfident in a super rude or like overconfident in a way that's like misleading people so way that's like misleading people so way that's like misleading people so they're they're all these tradeoffs um they're they're all these tradeoffs um they're they're all these tradeoffs um uh for example another thing is if there uh for example another thing is if there uh for example another thing is if there was a period during which models ours was a period during which models ours was a period during which models ours and I think others as well were T and I think others as well were T and I think others as well were T verbose right they would like repeat verbose right they would like repeat verbose right they would like repeat themselves they would say too much um

themselves they would say too much um themselves they would say too much um you can cut down on the verbosity by you can cut down on the verbosity by you can cut down on the verbosity by penalizing the models for for just penalizing the models for for just penalizing the models for for just talking for too long what happens when talking for too long what happens when talking for too long what happens when you do that if you do it in a crude way you do that if you do it in a crude way you do that if you do it in a crude way is when the models are coding sometimes is when the models are coding sometimes is when the models are coding sometimes they'll say of the code goes here right they'll say of the code goes here right they'll say of the code goes here right because they've learned that that's a because they've learned that that's a because they've learned that that's a way to economize and that they see it way to economize and that they see it way to economize and that they see it and then and then so that leads the and then and then so that leads the and then and then so that leads the model to be so-called lazy in coding model to be so-called lazy in coding model to be so-called lazy in coding where they where they where they're just where they where they where they're just where they where they where they're just like ah you can finish the rest of it like ah you can finish the rest of it like ah you can finish the rest of it it's not it's not because we want to you it's not it's not because we want to you it's not it's not because we want to you know save on compute or because you know know save on compute or because you know know save on compute or because you know the models are lazy and you know during the models are lazy and you know during the models are lazy and you know during winter break or any of the other kind of winter break or any of the other kind of winter break or any of the other kind of conspiracy theories that have that have conspiracy theories that have that have conspiracy theories that have that have that have come up it's actually it's that have come up it's actually it's that have come up it's actually it's just very hard to control the behavior just very hard to control the behavior just very hard to control the behavior of the model to steer the behavior of of the model to steer the behavior of of the model to steer the behavior of the model in all circum ances at once the model in all circum ances at once the model in all circum ances at once you can kind of there's this this whacka you can kind of there's this this whacka you can kind of there's this this whacka aspect where you push on one thing and aspect where you push on one thing and aspect where you push on one thing and like you know these these these you know like you know these these these you know like you know these these these you know these other things start to move as well these other things start to move as well these other things start to move as well that you may not even notice or measure that you may not even notice or measure that you may not even notice or measure and so one of the reasons that I that I and so one of the reasons that I that I and so one of the reasons that I that I care so much about uh you know kind of care so much about uh you know kind of care so much about uh you know kind of grand alignment of these AI systems in grand alignment of these AI systems in grand alignment of these AI systems in the future is actually these systems are the future is actually these systems are the future is actually these systems are actually quite unpredictable they're actually quite unpredictable they're actually quite unpredictable they're actually quite hard to steer and control actually quite hard to steer and control actually quite hard to steer and control um and this version we're seeing today um and this version we're seeing today um and this version we're seeing today of you make one thing better it makes of you make one thing better it makes of you make one thing better it makes another thing worse uh I think that's another thing worse uh I think that's another thing worse uh I think that's that's like a present day analog of that's like a present day analog of that's like a present day analog of future control problems in AI systems future control problems in AI systems future control problems in AI systems that we can start to study today right I that we can start to study today right I that we can start to study today right I think I think that that that difficulty

think I think that that that difficulty think I think that that that difficulty in in steering the behavior and in in in steering the behavior and in in in steering the behavior and in making sure that if we push an AI system making sure that if we push an AI system making sure that if we push an AI system in One Direction it doesn't push it in in One Direction it doesn't push it in in One Direction it doesn't push it in another Direction in some in some other another Direction in some in some other another Direction in some in some other ways that we didn't want uh I think ways that we didn't want uh I think ways that we didn't want uh I think that's that's kind of an that's kind of that's that's kind of an that's kind of that's that's kind of an that's kind of an early sign of things to come and if an early sign of things to come and if an early sign of things to come and if we can do a good job of solving this we can do a good job of solving this we can do a good job of solving this problem right of like you ask the model problem right of like you ask the model problem right of like you ask the model to like you know to like make and to like you know to like make and to like you know to like make and distribute small pox and it says no but distribute small pox and it says no but distribute small pox and it says no but it's willing to like help you in your it's willing to like help you in your it's willing to like help you in your graduate level virology class like how graduate level virology class like how graduate level virology class like how do we get both of those things at once do we get both of those things at once do we get both of those things at once it's hard it's very easy to go to one it's hard it's very easy to go to one it's hard it's very easy to go to one side or the other and it's a side or the other and it's a side or the other and it's a multi-dimensional problem and so uh I multi-dimensional problem and so uh I multi-dimensional problem and so uh I you know I think these questions of like you know I think these questions of like you know I think these questions of like shaping the models personality I think shaping the models personality I think shaping the models personality I think they're very hard I think we haven't they're very hard I think we haven't they're very hard I think we haven't done perfectly on them I think we've done perfectly on them I think we've done perfectly on them I think we've actually done the best of all the AI actually done the best of all the AI actually done the best of all the AI companies but still so far from perfect companies but still so far from perfect companies but still so far from perfect uh and I think if we can get this right uh and I think if we can get this right uh and I think if we can get this right if we can control the the you know if we can control the the you know if we can control the the you know control the false positives and false control the false positives and false control the false positives and false negatives in this this very kind of negatives in this this very kind of negatives in this this very kind of controlled present day environment will controlled present day environment will controlled present day environment will be much better at doing it for the be much better at doing it for the be much better at doing it for the future when our worry is you know will future when our worry is you know will future when our worry is you know will the models be super autonomous will they the models be super autonomous will they the models be super autonomous will they be able to you know make very dangerous be able to you know make very dangerous be able to you know make very dangerous things will they be able to autonomously things will they be able to autonomously things will they be able to autonomously you know build whole companies and are you know build whole companies and are you know build whole companies and are those companies aligned so so I I I those companies aligned so so I I I those companies aligned so so I I I think of this this present task as both think of this this present task as both think of this this present task as both vacine but also good practice for the vacine but also good practice for the vacine but also good practice for the future what's the current best way of future what's the current best way of future what's the current best way of gathering sort of user feedback like uh gathering sort of user feedback like uh gathering sort of user feedback like uh not anecdotal data but just large scale

not anecdotal data but just large scale not anecdotal data but just large scale data about pain points or the opposite data about pain points or the opposite data about pain points or the opposite of pain points positive things so on is of pain points positive things so on is of pain points positive things so on is it internal testing is it yeah A it internal testing is it yeah A it internal testing is it yeah A specific group testing a testing what specific group testing a testing what specific group testing a testing what what what works so so so typically um what what works so so so typically um what what works so so so typically um we'll have internal model bashings where we'll have internal model bashings where we'll have internal model bashings where all of anthropic anthropic is almost a all of anthropic anthropic is almost a all of anthropic anthropic is almost a thousand people um you know people just thousand people um you know people just thousand people um you know people just just try and break the model they try just try and break the model they try just try and break the model they try and interact with it various ways um uh and interact with it various ways um uh and interact with it various ways um uh we have a suite of evals uh for you know we have a suite of evals uh for you know we have a suite of evals uh for you know oh is the model refusing in ways that oh is the model refusing in ways that oh is the model refusing in ways that that it couldn't I think we even had a that it couldn't I think we even had a that it couldn't I think we even had a certainly eval because you know our our certainly eval because you know our our certainly eval because you know our our mod again at one point model had this mod again at one point model had this mod again at one point model had this problem where like it had this annoying problem where like it had this annoying problem where like it had this annoying tick where it would like respond to a tick where it would like respond to a tick where it would like respond to a wide range of questions by saying wide range of questions by saying wide range of questions by saying certainly I can help you with that certainly I can help you with that certainly I can help you with that certainly I would be happy to do that certainly I would be happy to do that certainly I would be happy to do that certainly this is correct um uh and so certainly this is correct um uh and so certainly this is correct um uh and so we had a like certainly eval which is we had a like certainly eval which is we had a like certainly eval which is like how how often does the model say like how how often does the model say like how how often does the model say certainly uh uh but but look this is certainly uh uh but but look this is certainly uh uh but but look this is just a whack-a-mole like like what if it just a whack-a-mole like like what if it just a whack-a-mole like like what if it switches from certainly to definitely switches from certainly to definitely switches from certainly to definitely like uh uh so you know every time we add like uh uh so you know every time we add like uh uh so you know every time we add a new eval and we're always evaluating a new eval and we're always evaluating a new eval and we're always evaluating for all the old things so we have for all the old things so we have for all the old things so we have hundreds of these evaluations but we hundreds of these evaluations but we hundreds of these evaluations but we find that there's no substitute for find that there's no substitute for find that there's no substitute for human interacting with it and so it's human interacting with it and so it's human interacting with it and so it's very much like the ordinary product very much like the ordinary product very much like the ordinary product development process we have like development process we have like development process we have like hundreds of people within anthropic bash hundreds of people within anthropic bash hundreds of people within anthropic bash the model then we do uh you know then we the model then we do uh you know then we the model then we do uh you know then we do external AB tests sometimes we'll run do external AB tests sometimes we'll run do external AB tests sometimes we'll run tests with contractors we pay tests with contractors we pay tests with contractors we pay contractors to interact with the model contractors to interact with the model contractors to interact with the model um so you put all of these things um so you put all of these things um so you put all of these things together and it's still not perfect you together and it's still not perfect you together and it's still not perfect you still see behaviors that you don't quite

still see behaviors that you don't quite still see behaviors that you don't quite want to see right you know you see you want to see right you know you see you want to see right you know you see you still see the model like refusing things still see the model like refusing things still see the model like refusing things that it just doesn't make sense to that it just doesn't make sense to that it just doesn't make sense to refuse um but I I I think trying to refuse um but I I I think trying to refuse um but I I I think trying to trying to solve this challenge right trying to solve this challenge right trying to solve this challenge right trying to stop the model from doing you trying to stop the model from doing you trying to stop the model from doing you know genuinely bad things that you know know genuinely bad things that you know know genuinely bad things that you know no one everyone agrees it shouldn't do no one everyone agrees it shouldn't do no one everyone agrees it shouldn't do right you know everyone everyone you right you know everyone everyone you right you know everyone everyone you know everyone agrees that you know the know everyone agrees that you know the know everyone agrees that you know the model shouldn't talk about you know I I model shouldn't talk about you know I I model shouldn't talk about you know I I don't know child abuse material right don't know child abuse material right don't know child abuse material right like everyone agrees the model shouldn't like everyone agrees the model shouldn't like everyone agrees the model shouldn't do that uh but but at the same time that do that uh but but at the same time that do that uh but but at the same time that it doesn't refuse in these dumb and it doesn't refuse in these dumb and it doesn't refuse in these dumb and stupid ways uh I think I think draw stupid ways uh I think I think draw stupid ways uh I think I think draw drawing that line as finely as possible drawing that line as finely as possible drawing that line as finely as possible approaching perfectly is still is still approaching perfectly is still is still approaching perfectly is still is still a challenge and we're getting better at a challenge and we're getting better at a challenge and we're getting better at it every day but there's there's a lot it every day but there's there's a lot it every day but there's there's a lot to be solved and again I would point to to be solved and again I would point to to be solved and again I would point to that as as an indicator of a challenge that as as an indicator of a challenge that as as an indicator of a challenge ahead in terms of steering much more ahead in terms of steering much more ahead in terms of steering much more powerful models do you think Claude 4.0 powerful models do you think Claude 4.0 powerful models do you think Claude 4.0 is ever coming out I don't want to is ever coming out I don't want to is ever coming out I don't want to commit to any naming scheme because if I commit to any naming scheme because if I commit to any naming scheme because if I say if I say here we're gonna have say if I say here we're gonna have say if I say here we're gonna have Claude 4 next year and then and then you Claude 4 next year and then and then you Claude 4 next year and then and then you know then we decide that like you know know then we decide that like you know know then we decide that like you know we should start over because there's a we should start over because there's a we should start over because there's a new type of mod like I I I I I I don't new type of mod like I I I I I I don't new type of mod like I I I I I I don't want to I don't want to commit to it I want to I don't want to commit to it I want to I don't want to commit to it I would expect in a normal course of would expect in a normal course of would expect in a normal course of business that Claude four would come business that Claude four would come business that Claude four would come after Claude 3.5 but but you know you after Claude 3.5 but but you know you after Claude 3.5 but but you know you you you never know in this wacky field you you never know in this wacky field you you never know in this wacky field right but the sort of this idea of right but the sort of this idea of right but the sort of this idea of scaling is continuing scal scaling is scaling is continuing scal scaling is scaling is continuing scal scaling is continuing there there will definitely continuing there there will definitely continuing there there will definitely be more powerful models coming from us be more powerful models coming from us be more powerful models coming from us in the models that exist today that is in the models that exist today that is in the models that exist today that is that is certain or if there if there that is certain or if there if there that is certain or if there if there aren't we've we've deeply failed as a

aren't we've we've deeply failed as a aren't we've we've deeply failed as a company okay can you explain the company okay can you explain the company okay can you explain the responsible scaling policy and the AI responsible scaling policy and the AI responsible scaling policy and the AI safety level standards ASL levels as safety level standards ASL levels as safety level standards ASL levels as much as I'm excited about the benefits much as I'm excited about the benefits much as I'm excited about the benefits of these models and you know we'll talk of these models and you know we'll talk of these models and you know we'll talk about that if we talk about Machines of about that if we talk about Machines of about that if we talk about Machines of Loving Grace um I'm I'm worried about Loving Grace um I'm I'm worried about Loving Grace um I'm I'm worried about the risk and I continue to be worried the risk and I continue to be worried the risk and I continue to be worried about the risks uh no one should think about the risks uh no one should think about the risks uh no one should think that you know Machines of loveing Grace that you know Machines of loveing Grace that you know Machines of loveing Grace was me me saying uh you know I'm no was me me saying uh you know I'm no was me me saying uh you know I'm no longer worried about the risks of these longer worried about the risks of these longer worried about the risks of these models I think they're two sides of the models I think they're two sides of the models I think they're two sides of the same coin the the uh Power of the models same coin the the uh Power of the models same coin the the uh Power of the models and their ability to solve all these and their ability to solve all these and their ability to solve all these problems in you know biology problems in you know biology problems in you know biology Neuroscience Economic Development Neuroscience Economic Development Neuroscience Economic Development government governance and peace large government governance and peace large government governance and peace large parts of the economy those those come parts of the economy those those come parts of the economy those those come with risks as well right with great with risks as well right with great with risks as well right with great power comes great responsibility right power comes great responsibility right power comes great responsibility right that's the the two are the two are that's the the two are the two are that's the the two are the two are paired uh things that are powerful can paired uh things that are powerful can paired uh things that are powerful can do good things and they can do bad do good things and they can do bad do good things and they can do bad things um I think of those risks as as things um I think of those risks as as things um I think of those risks as as being in you know several different being in you know several different being in you know several different different categories perhaps the two different categories perhaps the two different categories perhaps the two biggest risks that I think about and biggest risks that I think about and biggest risks that I think about and that's not to say that there aren't that's not to say that there aren't that's not to say that there aren't risks today that are that are important risks today that are that are important risks today that are that are important but when I think of the really the the but when I think of the really the the but when I think of the really the the you know the things that would happen on you know the things that would happen on you know the things that would happen on the grandest scale um one is what I call the grandest scale um one is what I call the grandest scale um one is what I call catastrophic misuse these are misuse of catastrophic misuse these are misuse of catastrophic misuse these are misuse of the models in domains like cyber bio the models in domains like cyber bio the models in domains like cyber bio radiological nuclear right things that radiological nuclear right things that radiological nuclear right things that could you know that could harm or even could you know that could harm or even could you know that could harm or even kill thousands even millions of people kill thousands even millions of people kill thousands even millions of people if they really really go wrong um like if they really really go wrong um like if they really really go wrong um like these are the you know number one these are the you know number one these are the you know number one priority to prevent and and here I would

priority to prevent and and here I would priority to prevent and and here I would just make a simple observation which is just make a simple observation which is just make a simple observation which is that Mo the models you know if if I look that Mo the models you know if if I look that Mo the models you know if if I look today at people who have done really bad today at people who have done really bad today at people who have done really bad things in the world um uh I think things in the world um uh I think things in the world um uh I think actually Humanity has been protected by actually Humanity has been protected by actually Humanity has been protected by the fact that the overlap between really the fact that the overlap between really the fact that the overlap between really smart well-educated people and people smart well-educated people and people smart well-educated people and people who want to do really horrific things who want to do really horrific things who want to do really horrific things has generally been small like you know has generally been small like you know has generally been small like you know let's say let's say I'm someone who you let's say let's say I'm someone who you let's say let's say I'm someone who you know uh you know I have a PhD in this know uh you know I have a PhD in this know uh you know I have a PhD in this field I have a well-paying job um field I have a well-paying job um field I have a well-paying job um there's so much to lose why do I want to there's so much to lose why do I want to there's so much to lose why do I want to like you know even even assuming I'm like you know even even assuming I'm like you know even even assuming I'm completely evil which which most people completely evil which which most people completely evil which which most people are not um why why you know why would are not um why why you know why would are not um why why you know why would such a person risk their risk their you such a person risk their risk their you such a person risk their risk their you know risk their life RK risk their their know risk their life RK risk their their know risk their life RK risk their their legacy their reputation to to do legacy their reputation to to do legacy their reputation to to do something like you know truly truly evil something like you know truly truly evil something like you know truly truly evil if we had a lot more people like that if we had a lot more people like that if we had a lot more people like that the world would be a much more dangerous the world would be a much more dangerous the world would be a much more dangerous place and so my my My worry is that by place and so my my My worry is that by place and so my my My worry is that by being a a much more intelligent agent AI being a a much more intelligent agent AI being a a much more intelligent agent AI could break that correlation and so I I could break that correlation and so I I could break that correlation and so I I I I I do have serious worries about that I I I do have serious worries about that I I I do have serious worries about that I believe we can prevent those worries I believe we can prevent those worries I believe we can prevent those worries uh but you know I I think as a uh but you know I I think as a uh but you know I I think as a Counterpoint to Machines of Loving Grace Counterpoint to Machines of Loving Grace Counterpoint to Machines of Loving Grace I want to say that this is I there's I want to say that this is I there's I want to say that this is I there's still serious risks and and the second still serious risks and and the second still serious risks and and the second range of risks would be the autonomy range of risks would be the autonomy range of risks would be the autonomy risks which is the idea that models risks which is the idea that models risks which is the idea that models might on their own particularly as we might on their own particularly as we might on their own particularly as we give them more agency than they've had give them more agency than they've had give them more agency than they've had in the past uh particularly as we give in the past uh particularly as we give in the past uh particularly as we give them supervision over wider tasks like them supervision over wider tasks like them supervision over wider tasks like you know writing whole code bases or

you know writing whole code bases or you know writing whole code bases or someday even you know effectively someday even you know effectively someday even you know effectively operating entire entire companies operating entire entire companies operating entire entire companies they're on a long enough leash are they they're on a long enough leash are they they're on a long enough leash are they are they doing what we really want them are they doing what we really want them are they doing what we really want them to do it's very difficult to even to do it's very difficult to even to do it's very difficult to even understand in detail what they're doing understand in detail what they're doing understand in detail what they're doing let alone let alone control it and like let alone let alone control it and like let alone let alone control it and like I said this these early signs that it's I said this these early signs that it's I said this these early signs that it's it's hard to perfectly draw the boundary it's hard to perfectly draw the boundary it's hard to perfectly draw the boundary between things the model should do and between things the model should do and between things the model should do and things the model shouldn't do that that things the model shouldn't do that that things the model shouldn't do that that you know if if you go to one side you you know if if you go to one side you you know if if you go to one side you get things that are annoying and useless get things that are annoying and useless get things that are annoying and useless and you go to the other side you get and you go to the other side you get and you go to the other side you get other behaviors if you fix one thing it other behaviors if you fix one thing it other behaviors if you fix one thing it creates other problems we're getting creates other problems we're getting creates other problems we're getting better and better at solving this I better and better at solving this I better and better at solving this I don't think this is an unsolvable don't think this is an unsolvable don't think this is an unsolvable problem I think this is a you know this problem I think this is a you know this problem I think this is a you know this is a science like like the safety of is a science like like the safety of is a science like like the safety of airplanes or the safety of cars or the airplanes or the safety of cars or the airplanes or the safety of cars or the safety of drugs I you know I I don't safety of drugs I you know I I don't safety of drugs I you know I I don't think there's any big thing we're think there's any big thing we're think there's any big thing we're missing I just think we need to get missing I just think we need to get missing I just think we need to get better at controlling these models and better at controlling these models and better at controlling these models and so these are these are the two risks I'm so these are these are the two risks I'm so these are these are the two risks I'm worried about and our responsible worried about and our responsible worried about and our responsible scaling plan which I'll recognize is a scaling plan which I'll recognize is a scaling plan which I'll recognize is a very long-winded answer to your question very long-winded answer to your question very long-winded answer to your question I love it I love it our responsible I love it I love it our responsible I love it I love it our responsible scaling plan is designed to address scaling plan is designed to address scaling plan is designed to address these two types of risks and so every these two types of risks and so every these two types of risks and so every time we develop a new model we basically time we develop a new model we basically time we develop a new model we basically test it for its ability to do both of test it for its ability to do both of test it for its ability to do both of these bad things so if I were to back up these bad things so if I were to back up these bad things so if I were to back up a little bit um I I think we have a I a little bit um I I think we have a I a little bit um I I think we have a I think we have an interesting dilemma think we have an interesting dilemma think we have an interesting dilemma with AI systems where they're not yet with AI systems where they're not yet with AI systems where they're not yet powerful enough to present these powerful enough to present these powerful enough to present these catastrophes I don't know that I don't catastrophes I don't know that I don't catastrophes I don't know that I don't know they'll ever present prevent these know they'll ever present prevent these know they'll ever present prevent these catastrophes it's possible they won't

catastrophes it's possible they won't catastrophes it's possible they won't but the the case for worry the case for but the the case for worry the case for but the the case for worry the case for risk is strong enough that we should we risk is strong enough that we should we risk is strong enough that we should we should act now and and they're they're should act now and and they're they're should act now and and they're they're getting better very very fast right I getting better very very fast right I getting better very very fast right I you know I testified in the Senate that you know I testified in the Senate that you know I testified in the Senate that you know we might have serious bio risks you know we might have serious bio risks you know we might have serious bio risks within two to three years that was about within two to three years that was about within two to three years that was about a year ago things have preceded preceded a year ago things have preceded preceded a year ago things have preceded preceded a pace uh uh so we have this thing where a pace uh uh so we have this thing where a pace uh uh so we have this thing where it's like it's it's it's surprisingly it's like it's it's it's surprisingly it's like it's it's it's surprisingly hard to to address these risks because hard to to address these risks because hard to to address these risks because they're not here today they don't exist they're not here today they don't exist they're not here today they don't exist they're like ghosts but they're coming they're like ghosts but they're coming they're like ghosts but they're coming at us so fast because the models are at us so fast because the models are at us so fast because the models are improving so fast so so how do you deal improving so fast so so how do you deal improving so fast so so how do you deal with something that's not here today with something that's not here today with something that's not here today doesn't exist but is is coming at us doesn't exist but is is coming at us doesn't exist but is is coming at us very fast uh so the solution we came up very fast uh so the solution we came up very fast uh so the solution we came up with for that in in collaboration with with for that in in collaboration with with for that in in collaboration with uh you know people like uh the uh you know people like uh the uh you know people like uh the organization meter and Paul Christiano organization meter and Paul Christiano organization meter and Paul Christiano is okay what what what what you need for is okay what what what what you need for is okay what what what what you need for that or you need tests to tell you when that or you need tests to tell you when that or you need tests to tell you when the risk is getting close you need an the risk is getting close you need an the risk is getting close you need an early warning system and and so every early warning system and and so every early warning system and and so every time we have uh a new model we test it time we have uh a new model we test it time we have uh a new model we test it for it capability to do these cbrn tasks for it capability to do these cbrn tasks for it capability to do these cbrn tasks as well as testing it for you know how as well as testing it for you know how as well as testing it for you know how capable it is of doing tasks capable it is of doing tasks capable it is of doing tasks autonomously on its own and uh in the autonomously on its own and uh in the autonomously on its own and uh in the latest version of our RSP which we latest version of our RSP which we latest version of our RSP which we released in the last in the last month released in the last in the last month released in the last in the last month or two uh the way we test autonomy risks or two uh the way we test autonomy risks or two uh the way we test autonomy risks is the model the the AI model's ability is the model the the AI model's ability is the model the the AI model's ability to do aspects of AI research itself uh to do aspects of AI research itself uh to do aspects of AI research itself uh which when the model when the AI models which when the model when the AI models which when the model when the AI models can do AI research they become kind of

can do AI research they become kind of can do AI research they become kind of truly truly autonomous on and that you truly truly autonomous on and that you truly truly autonomous on and that you know that threshold is important for a know that threshold is important for a know that threshold is important for a bunch of other ways and and so what do bunch of other ways and and so what do bunch of other ways and and so what do we then do with these tasks the RSP we then do with these tasks the RSP we then do with these tasks the RSP basically develops what we've called an basically develops what we've called an basically develops what we've called an if then structure which is if the models if then structure which is if the models if then structure which is if the models pass a certain capability then we impose pass a certain capability then we impose pass a certain capability then we impose a certain set of Safety and Security a certain set of Safety and Security a certain set of Safety and Security requirements on them so today's models requirements on them so today's models requirements on them so today's models are what's called are what's called are what's called asl2 models that were a asl1 is for asl2 models that were a asl1 is for asl2 models that were a asl1 is for systems that manifestly don't pose any systems that manifestly don't pose any systems that manifestly don't pose any risk of autonomy or misuse so for risk of autonomy or misuse so for risk of autonomy or misuse so for example a chess plane bot deep blue example a chess plane bot deep blue example a chess plane bot deep blue would be asl1 it's just manifestly the would be asl1 it's just manifestly the would be asl1 it's just manifestly the case that you can't use deep blue for case that you can't use deep blue for case that you can't use deep blue for anything other than chess it was just anything other than chess it was just anything other than chess it was just designed for chess no one's going to use designed for chess no one's going to use designed for chess no one's going to use it to like you know to conduct a it to like you know to conduct a it to like you know to conduct a masterful Cyber attack or to you know masterful Cyber attack or to you know masterful Cyber attack or to you know run wild and take over the world asl2 is run wild and take over the world asl2 is run wild and take over the world asl2 is today's AI systems where we've measured today's AI systems where we've measured today's AI systems where we've measured them and we think these systems are them and we think these systems are them and we think these systems are simply not smart enough to uh to you simply not smart enough to uh to you simply not smart enough to uh to you know autonomously self-replicate or know autonomously self-replicate or know autonomously self-replicate or conduct a bunch of tasks uh and also not conduct a bunch of tasks uh and also not conduct a bunch of tasks uh and also not smart enough to provide meaningful smart enough to provide meaningful smart enough to provide meaningful information about cbrn risks and how to information about cbrn risks and how to information about cbrn risks and how to build cbrn weapons above and beyond what build cbrn weapons above and beyond what build cbrn weapons above and beyond what can be known from looking at Google uh can be known from looking at Google uh can be known from looking at Google uh in fact sometimes they do provide in fact sometimes they do provide in fact sometimes they do provide information but but not above and beyond information but but not above and beyond information but but not above and beyond a search engine but not in a way that a search engine but not in a way that a search engine but not in a way that can be stitched together um not not in a can be stitched together um not not in a can be stitched together um not not in a way that kind of end to end is dangerous

way that kind of end to end is dangerous way that kind of end to end is dangerous enough so enough so enough so asl3 is going to be the point at which asl3 is going to be the point at which asl3 is going to be the point at which uh the models are helpful enough to uh the models are helpful enough to uh the models are helpful enough to enhance the capabilities of non-state enhance the capabilities of non-state enhance the capabilities of non-state actors right State actors can already do actors right State actors can already do actors right State actors can already do a lot a lot of unfortunately to a high a lot a lot of unfortunately to a high a lot a lot of unfortunately to a high level of proficiency a lot of these very level of proficiency a lot of these very level of proficiency a lot of these very dangerous and destructive things the dangerous and destructive things the dangerous and destructive things the difference is that non-state non-state difference is that non-state non-state difference is that non-state non-state actors are not capable of it and so when actors are not capable of it and so when actors are not capable of it and so when we get to asl3 we'll take special we get to asl3 we'll take special we get to asl3 we'll take special security precautions designed to be be security precautions designed to be be security precautions designed to be be sufficient to prevent theft of the model sufficient to prevent theft of the model sufficient to prevent theft of the model by non-state actors and misuse of the by non-state actors and misuse of the by non-state actors and misuse of the model as it's deployed uh will have to model as it's deployed uh will have to model as it's deployed uh will have to have enhanced filters targeted at these have enhanced filters targeted at these have enhanced filters targeted at these particular areas cyber bio nuclear cyber particular areas cyber bio nuclear cyber particular areas cyber bio nuclear cyber bio nuclear and model autonomy Which is bio nuclear and model autonomy Which is bio nuclear and model autonomy Which is less a misuse risk and more a risk of less a misuse risk and more a risk of less a misuse risk and more a risk of the model doing bad things itself asl4 the model doing bad things itself asl4 the model doing bad things itself asl4 getting to the point where these models getting to the point where these models getting to the point where these models could could enhance the capability of a could could enhance the capability of a could could enhance the capability of a of a of a all knowledgeable State actor of a of a all knowledgeable State actor of a of a all knowledgeable State actor Andor become the you know the main Andor become the you know the main Andor become the you know the main source of such a risk like if you wanted source of such a risk like if you wanted source of such a risk like if you wanted to engage in such a risk the main way to engage in such a risk the main way to engage in such a risk the main way you would do it is through a model and you would do it is through a model and you would do it is through a model and then I think asl4 on the autonomy side then I think asl4 on the autonomy side then I think asl4 on the autonomy side it's it's some some some amount of it's it's some some some amount of it's it's some some some amount of acceleration in AI research capabilities acceleration in AI research capabilities acceleration in AI research capabilities with an with an AI model and then asl5 with an with an AI model and then asl5 with an with an AI model and then asl5 is where we would get to the models that is where we would get to the models that is where we would get to the models that are you know that are that are kind of are you know that are that are kind of are you know that are that are kind of that are kind of you know truly capable that are kind of you know truly capable that are kind of you know truly capable that it could exceed Humanity in their that it could exceed Humanity in their that it could exceed Humanity in their ability to do to do any of these tasks ability to do to do any of these tasks ability to do to do any of these tasks and so the the the point of the if then

and so the the the point of the if then and so the the the point of the if then structure commitment is is basically to structure commitment is is basically to structure commitment is is basically to say say say look I don't know I've been I've been look I don't know I've been I've been look I don't know I've been I've been working with these models for many years working with these models for many years working with these models for many years and I've been worried about risk for and I've been worried about risk for and I've been worried about risk for many years it's actually kind of many years it's actually kind of many years it's actually kind of dangerous to cry wolf it's actually kind dangerous to cry wolf it's actually kind dangerous to cry wolf it's actually kind of dangerous to say this you know this of dangerous to say this you know this of dangerous to say this you know this this model is this model is risky and this model is this model is risky and this model is this model is risky and you know people look at it and they say you know people look at it and they say you know people look at it and they say this is manifestly not dangerous again this is manifestly not dangerous again this is manifestly not dangerous again it's it's it's the the delicacy of the it's it's it's the the delicacy of the it's it's it's the the delicacy of the risk isn't here to today but it's coming risk isn't here to today but it's coming risk isn't here to today but it's coming at us fast how do you deal with that at us fast how do you deal with that at us fast how do you deal with that it's it's really vexing to a risk it's it's really vexing to a risk it's it's really vexing to a risk planner to deal with it and so this if planner to deal with it and so this if planner to deal with it and so this if then structure basically says look we then structure basically says look we then structure basically says look we don't want to antagonize a bunch of don't want to antagonize a bunch of don't want to antagonize a bunch of people we don't want to harm our own you people we don't want to harm our own you people we don't want to harm our own you know our our kind of own ability to have know our our kind of own ability to have know our our kind of own ability to have a place in the conversation by imposing a place in the conversation by imposing a place in the conversation by imposing these these these these very honorous burdens on models these very honorous burdens on models these very honorous burdens on models that are not dangerous today so the if that are not dangerous today so the if that are not dangerous today so the if then the trigger commitment is basically then the trigger commitment is basically then the trigger commitment is basically a way to deal with this says you claim a way to deal with this says you claim a way to deal with this says you claim clamp down hard when you can show that clamp down hard when you can show that clamp down hard when you can show that the model is dangerous and of course the model is dangerous and of course the model is dangerous and of course what has to come with that is you know what has to come with that is you know what has to come with that is you know enough of a buffer threshold that that enough of a buffer threshold that that enough of a buffer threshold that that you know you can you can uh you know you know you can you can uh you know you know you can you can uh you know you're you're you're you're not at high you're you're you're you're not at high you're you're you're you're not at high risk of kind of missing the danger it's risk of kind of missing the danger it's risk of kind of missing the danger it's not a perfect framework we've had to not a perfect framework we've had to not a perfect framework we've had to change it every every uh you know we change it every every uh you know we change it every every uh you know we came out with a new one just a few weeks came out with a new one just a few weeks came out with a new one just a few weeks ago and probably probably going forward ago and probably probably going forward ago and probably probably going forward we might release new ones multiple times we might release new ones multiple times we might release new ones multiple times a year because it's it's hard to get a year because it's it's hard to get a year because it's it's hard to get these policies right like technically these policies right like technically these policies right like technically organizationally from a research organizationally from a research organizationally from a research perspective but that is the proposal if perspective but that is the proposal if perspective but that is the proposal if then commitments and triggers in order

then commitments and triggers in order then commitments and triggers in order to minimize burdens and false alarms now to minimize burdens and false alarms now to minimize burdens and false alarms now but really react appropriately when the but really react appropriately when the but really react appropriately when the dangers are here what do you think the dangers are here what do you think the dangers are here what do you think the timeline for asl3 is where several of timeline for asl3 is where several of timeline for asl3 is where several of the triggers are fired and what do you the triggers are fired and what do you the triggers are fired and what do you think the timeline is for asl4 yeah so think the timeline is for asl4 yeah so think the timeline is for asl4 yeah so that is hotly debated within the company that is hotly debated within the company that is hotly debated within the company um uh we are working actively to prepare um uh we are working actively to prepare um uh we are working actively to prepare asl3 uh security uh security measures as asl3 uh security uh security measures as asl3 uh security uh security measures as well as ASL three deployment measures um well as ASL three deployment measures um well as ASL three deployment measures um I'm not going to go into detail but I'm not going to go into detail but I'm not going to go into detail but we've made we've made a lot of progress we've made we've made a lot of progress we've made we've made a lot of progress on both and you know we're we're on both and you know we're we're on both and you know we're we're prepared to be I think ready quite soon prepared to be I think ready quite soon prepared to be I think ready quite soon uh I would I would not be surpris I uh I would I would not be surpris I uh I would I would not be surpris I would not be surprised at all if we hit would not be surprised at all if we hit would not be surprised at all if we hit ASL 3 uh next year there was some ASL 3 uh next year there was some ASL 3 uh next year there was some concern that we we might even hit it uh concern that we we might even hit it uh concern that we we might even hit it uh uh this year that's still that's still uh this year that's still that's still uh this year that's still that's still possible that could still happen it's possible that could still happen it's possible that could still happen it's like very hard to say but like I would like very hard to say but like I would like very hard to say but like I would be very very surprised if it was like be very very surprised if it was like be very very surprised if it was like 2030 uh I think it's much sooner than 2030 uh I think it's much sooner than 2030 uh I think it's much sooner than that so there's a protocols for that so there's a protocols for that so there's a protocols for detecting it the if then and then detecting it the if then and then detecting it the if then and then there's protocols for how to respond to there's protocols for how to respond to there's protocols for how to respond to it yes how difficult is the second the it yes how difficult is the second the it yes how difficult is the second the ladder yeah I think for asl3 it's ladder yeah I think for asl3 it's ladder yeah I think for asl3 it's primarily about security um and and primarily about security um and and primarily about security um and and about you know filters on the model about you know filters on the model about you know filters on the model relating to a very narrow set of areas relating to a very narrow set of areas relating to a very narrow set of areas when we deploy the model because at asl3 when we deploy the model because at asl3 when we deploy the model because at asl3 the model isn't autonomous yet um uh and the model isn't autonomous yet um uh and the model isn't autonomous yet um uh and and so you don't have to worry about you and so you don't have to worry about you and so you don't have to worry about you know kind of the model itself behaving know kind of the model itself behaving know kind of the model itself behaving in a bad way even when it's deployed in a bad way even when it's deployed in a bad way even when it's deployed internally so I think the asl3 measures

internally so I think the asl3 measures internally so I think the asl3 measures are are I won't say straightforward are are I won't say straightforward are are I won't say straightforward they're they're they're they're rigorous they're they're they're they're rigorous they're they're they're they're rigorous but they're easier to reason about I but they're easier to reason about I but they're easier to reason about I think once we get to think once we get to think once we get to asl4 um we start to have worries about asl4 um we start to have worries about asl4 um we start to have worries about the models being smart enough that they the models being smart enough that they the models being smart enough that they might sandbag tests they might not tell might sandbag tests they might not tell might sandbag tests they might not tell the truth about tests um we had some the truth about tests um we had some the truth about tests um we had some results came out about like sleeper results came out about like sleeper results came out about like sleeper agents and there was a more recent paper agents and there was a more recent paper agents and there was a more recent paper about you know can can the models uh uh about you know can can the models uh uh about you know can can the models uh uh mislead attempts to you know s sandbag mislead attempts to you know s sandbag mislead attempts to you know s sandbag their own abilities right show them you their own abilities right show them you their own abilities right show them you know uh uh present themselves as being know uh uh present themselves as being know uh uh present themselves as being less capable than they are and so I less capable than they are and so I less capable than they are and so I think with asl4 there's going to be an think with asl4 there's going to be an think with asl4 there's going to be an important component of using other important component of using other important component of using other things than just interacting with the things than just interacting with the things than just interacting with the models for example interpretability or models for example interpretability or models for example interpretability or hidden chains of thought uh where you hidden chains of thought uh where you hidden chains of thought uh where you have to look inside the model and verify have to look inside the model and verify have to look inside the model and verify via some other mechanism that that is via some other mechanism that that is via some other mechanism that that is not you know is not as easily corrupted not you know is not as easily corrupted not you know is not as easily corrupted as what the model says as what the model says as what the model says uh that that you know that that that the uh that that you know that that that the uh that that you know that that that the model indeed has some property uh so model indeed has some property uh so model indeed has some property uh so we're still working on asl4 one of the we're still working on asl4 one of the we're still working on asl4 one of the properties of the RSP is that we we properties of the RSP is that we we properties of the RSP is that we we don't specify asl4 until we've hit ASL 3 don't specify asl4 until we've hit ASL 3 don't specify asl4 until we've hit ASL 3 be and and I think that's proven to be a be and and I think that's proven to be a be and and I think that's proven to be a wise decision because even with asl3 it wise decision because even with asl3 it wise decision because even with asl3 it again it's hard to know this stuff in again it's hard to know this stuff in again it's hard to know this stuff in detail and and it it we want to take as detail and and it it we want to take as detail and and it it we want to take as much time as we can possibly take to get much time as we can possibly take to get much time as we can possibly take to get these things right so for asl3 the bad these things right so for asl3 the bad these things right so for asl3 the bad actor will be the humans humans yes and actor will be the humans humans yes and actor will be the humans humans yes and so there it's a little bit more uh for so there it's a little bit more uh for so there it's a little bit more uh for asl4 it's both I think it's both and so

asl4 it's both I think it's both and so asl4 it's both I think it's both and so deception and that's where mechanistic deception and that's where mechanistic deception and that's where mechanistic interpretability comes into play and interpretability comes into play and interpretability comes into play and hopefully the techniques used for that hopefully the techniques used for that hopefully the techniques used for that are not made accessible to the model are not made accessible to the model are not made accessible to the model yeah I mean of course you can hook up yeah I mean of course you can hook up yeah I mean of course you can hook up the mechanistic contribut ability to the the mechanistic contribut ability to the the mechanistic contribut ability to the model itself um but then You' then then model itself um but then You' then then model itself um but then You' then then you then you've kind of lost it as a you then you've kind of lost it as a you then you've kind of lost it as a reliable indicator of uh of uh of of of reliable indicator of uh of uh of of of reliable indicator of uh of uh of of of the model State there are a bunch of the model State there are a bunch of the model State there are a bunch of exotic ways you can think of that it exotic ways you can think of that it exotic ways you can think of that it might also not be reliable like if the might also not be reliable like if the might also not be reliable like if the you know model gets smart enough that it you know model gets smart enough that it you know model gets smart enough that it can like you know jump computers and can like you know jump computers and can like you know jump computers and like read the code where you're like like read the code where you're like like read the code where you're like looking at its internal State we've looking at its internal State we've looking at its internal State we've thought about some of those I think thought about some of those I think thought about some of those I think they're exotic enough there are ways to they're exotic enough there are ways to they're exotic enough there are ways to render them unlikely but yeah generally render them unlikely but yeah generally render them unlikely but yeah generally you want to you want to preserve you want to you want to preserve you want to you want to preserve mechanistic interpretability as a kind mechanistic interpretability as a kind mechanistic interpretability as a kind of verification set or test set that's of verification set or test set that's of verification set or test set that's separate from the training process of separate from the training process of separate from the training process of the model see I think uh as these models the model see I think uh as these models the model see I think uh as these models become better and better conversation become better and better conversation become better and better conversation and become smarter social engineering and become smarter social engineering and become smarter social engineering becomes a threat too cuz they oh yeah becomes a threat too cuz they oh yeah becomes a threat too cuz they oh yeah that can start being very convincing to that can start being very convincing to that can start being very convincing to the engineers inside companies oh yeah the engineers inside companies oh yeah the engineers inside companies oh yeah yeah it's actually like you know we've yeah it's actually like you know we've yeah it's actually like you know we've we've seen lots of examples of we've seen lots of examples of we've seen lots of examples of demagoguery in our life from humans and demagoguery in our life from humans and demagoguery in our life from humans and and you know there's a concern that and you know there's a concern that and you know there's a concern that models could do that could do that as models could do that could do that as models could do that could do that as well one of the ways that cloud has been well one of the ways that cloud has been well one of the ways that cloud has been getting more and more powerful is it's getting more and more powerful is it's getting more and more powerful is it's now able to do some agentic stuff um now able to do some agentic stuff um now able to do some agentic stuff um computer use uh there's also an analysis computer use uh there's also an analysis computer use uh there's also an analysis within the sandbox of claw. a itself but within the sandbox of claw. a itself but within the sandbox of claw. a itself but let's talk about computer use that's let's talk about computer use that's let's talk about computer use that's seems to me super exciting that you can seems to me super exciting that you can seems to me super exciting that you can just give Claude a task and it uh takes just give Claude a task and it uh takes just give Claude a task and it uh takes a bunch of actions figures it out and a bunch of actions figures it out and a bunch of actions figures it out and has access to the your computer through

has access to the your computer through has access to the your computer through screenshots so can you explain how that screenshots so can you explain how that screenshots so can you explain how that works uh and where that's headed yeah works uh and where that's headed yeah works uh and where that's headed yeah it's actually relatively simple so it's actually relatively simple so it's actually relatively simple so Claude has has had for a long time since Claude has has had for a long time since Claude has has had for a long time since since Claude 3 back in March the ability since Claude 3 back in March the ability since Claude 3 back in March the ability to analyze images and respond to them to analyze images and respond to them to analyze images and respond to them with text the the only new thing we with text the the only new thing we with text the the only new thing we added is those images can be screenshot added is those images can be screenshot added is those images can be screenshot shots of a computer and in response we shots of a computer and in response we shots of a computer and in response we train the model to give a location on train the model to give a location on train the model to give a location on the screen where you can click Andor the screen where you can click Andor the screen where you can click Andor buttons on the keyboard you can press in buttons on the keyboard you can press in buttons on the keyboard you can press in order to take action and it turns out order to take action and it turns out order to take action and it turns out that with actually not all that much that with actually not all that much that with actually not all that much additional training the models can get additional training the models can get additional training the models can get quite good at that task it's a good quite good at that task it's a good quite good at that task it's a good example of generalization um you know example of generalization um you know example of generalization um you know people sometimes say if you get to low people sometimes say if you get to low people sometimes say if you get to low earth orbit you're like halfway to earth orbit you're like halfway to earth orbit you're like halfway to anywhere right because of how much it anywhere right because of how much it anywhere right because of how much it takes to escape the gravity well if you takes to escape the gravity well if you takes to escape the gravity well if you have a strong pre-trained model I feel have a strong pre-trained model I feel have a strong pre-trained model I feel like you're halfway to anywhere uh in like you're halfway to anywhere uh in like you're halfway to anywhere uh in ter in terms of in terms of the ter in terms of in terms of the ter in terms of in terms of the intelligence space uh uh uh and and and intelligence space uh uh uh and and and intelligence space uh uh uh and and and so actually it didn't it didn't take all so actually it didn't it didn't take all so actually it didn't it didn't take all that much to get to get Claude to do that much to get to get Claude to do that much to get to get Claude to do this and you can just set that in a loop this and you can just set that in a loop this and you can just set that in a loop give the model a screenshot tell it what give the model a screenshot tell it what give the model a screenshot tell it what to click on give it the next screenshot to click on give it the next screenshot to click on give it the next screenshot tell it what to click on and and that tell it what to click on and and that tell it what to click on and and that turns into a full kind of almost almost turns into a full kind of almost almost turns into a full kind of almost almost 3D video interaction of the model and 3D video interaction of the model and 3D video interaction of the model and it's able to do all of these tasks right it's able to do all of these tasks right it's able to do all of these tasks right you know we we showed these demos where you know we we showed these demos where you know we we showed these demos where it's able to like fill out spreadsheets it's able to like fill out spreadsheets it's able to like fill out spreadsheets it's able to kind of like interact with it's able to kind of like interact with it's able to kind of like interact with a website it's able to you know um you a website it's able to you know um you a website it's able to you know um you know it's able to open all kinds of you know it's able to open all kinds of you know it's able to open all kinds of you know programs different operating know programs different operating know programs different operating systems Windows Linux Mac uh uh so uh

systems Windows Linux Mac uh uh so uh systems Windows Linux Mac uh uh so uh you know I think all of that is very you know I think all of that is very you know I think all of that is very exciting I I will say while in theory exciting I I will say while in theory exciting I I will say while in theory there's nothing you could do there that there's nothing you could do there that there's nothing you could do there that you couldn't have done through just you couldn't have done through just you couldn't have done through just giving the model the API to drive the giving the model the API to drive the giving the model the API to drive the computer screen uh this really lowers computer screen uh this really lowers computer screen uh this really lowers the barrier and you know there's there's the barrier and you know there's there's the barrier and you know there's there's there's a lot of folks who who who there's a lot of folks who who who there's a lot of folks who who who either you know kind of kind of ar ar either you know kind of kind of ar ar either you know kind of kind of ar ar you know aren't in a position to to you know aren't in a position to to you know aren't in a position to to interact with those apis or it takes interact with those apis or it takes interact with those apis or it takes them a long time to do it's just the them a long time to do it's just the them a long time to do it's just the screen is just a universal interface screen is just a universal interface screen is just a universal interface that's a lot easier to interact with and that's a lot easier to interact with and that's a lot easier to interact with and so I expect over time this is going to so I expect over time this is going to so I expect over time this is going to lower a bunch of barriers now honestly lower a bunch of barriers now honestly lower a bunch of barriers now honestly the current model has there's there it the current model has there's there it the current model has there's there it leaves a lot still to be desired and we leaves a lot still to be desired and we leaves a lot still to be desired and we were we were honest about that in the were we were honest about that in the were we were honest about that in the blog right it makes mistakes it blog right it makes mistakes it blog right it makes mistakes it misclicks and we we you know we were misclicks and we we you know we were misclicks and we we you know we were careful to warn people hey this thing careful to warn people hey this thing careful to warn people hey this thing isn't you can't just leave this thing to isn't you can't just leave this thing to isn't you can't just leave this thing to you know run on your computer for you know run on your computer for you know run on your computer for minutes and minutes um you got to give minutes and minutes um you got to give minutes and minutes um you got to give this thing boundaries and guard rails this thing boundaries and guard rails this thing boundaries and guard rails and I think that's one of the reasons we and I think that's one of the reasons we and I think that's one of the reasons we released it first in an API form rather released it first in an API form rather released it first in an API form rather than kind of you know this this kind of than kind of you know this this kind of than kind of you know this this kind of just just hand it just hand it to the just just hand it just hand it to the just just hand it just hand it to the consumer and give it control of their of consumer and give it control of their of consumer and give it control of their of their of their of their computer um but their of their of their computer um but their of their of their computer um but but you know I definitely feel that it's but you know I definitely feel that it's but you know I definitely feel that it's important to get these capabilities out important to get these capabilities out important to get these capabilities out there as models get more powerful we're there as models get more powerful we're there as models get more powerful we're going to have to Grapple with you know going to have to Grapple with you know going to have to Grapple with you know how do we use these capabilities safely how do we use these capabilities safely how do we use these capabilities safely how do we prevent them from being abused how do we prevent them from being abused how do we prevent them from being abused uh and and you know I think I think uh and and you know I think I think uh and and you know I think I think releasing releasing the model while releasing releasing the model while releasing releasing the model while while while the capabilities are are you while while the capabilities are are you while while the capabilities are are you know are are still are still limited is know are are still are still limited is know are are still are still limited is is is very helpful in terms of in terms is is very helpful in terms of in terms is is very helpful in terms of in terms of doing that um you know I think since of doing that um you know I think since of doing that um you know I think since it's been released a number of customers

it's been released a number of customers it's been released a number of customers I think uh repet was maybe was maybe one I think uh repet was maybe was maybe one I think uh repet was maybe was maybe one of the the the most uh uh quickest of the the the most uh uh quickest of the the the most uh uh quickest quickest quickest to quickest to deploy quickest quickest to quickest to deploy quickest quickest to quickest to deploy things um have have you know have made things um have have you know have made things um have have you know have made use of it in various ways people have use of it in various ways people have use of it in various ways people have hooked up demos for you know Windows hooked up demos for you know Windows hooked up demos for you know Windows desktops Macs desktops Macs desktops Macs uh uh you know Linux Linux machines uh uh uh you know Linux Linux machines uh uh uh you know Linux Linux machines uh so yeah it's been it's been it's been so yeah it's been it's been it's been so yeah it's been it's been it's been very exciting I think as with as with very exciting I think as with as with very exciting I think as with as with anything else you know it it it comes anything else you know it it it comes anything else you know it it it comes with new exciting abilities and then with new exciting abilities and then with new exciting abilities and then then then you know then then with those then then you know then then with those then then you know then then with those new exciting abilities we have to think new exciting abilities we have to think new exciting abilities we have to think about how to how to you know make the about how to how to you know make the about how to how to you know make the model you know safe reliable do what model you know safe reliable do what model you know safe reliable do what humans want them to do I mean it's the humans want them to do I mean it's the humans want them to do I mean it's the same it's the same story for everything same it's the same story for everything same it's the same story for everything right same thing it's that same tension right same thing it's that same tension right same thing it's that same tension but but the possibility of use cases but but the possibility of use cases but but the possibility of use cases here is just the the range is incredible here is just the the range is incredible here is just the the range is incredible so uh how much to make it work really so uh how much to make it work really so uh how much to make it work really well in the future how much do you have well in the future how much do you have well in the future how much do you have to specially kind of uh go beyond what's to specially kind of uh go beyond what's to specially kind of uh go beyond what's the pre-trained models doing do more the pre-trained models doing do more the pre-trained models doing do more posttraining rhf or supervised posttraining rhf or supervised posttraining rhf or supervised fine-tuning or synthetic data just for fine-tuning or synthetic data just for fine-tuning or synthetic data just for the agent stff yeah I think speaking at the agent stff yeah I think speaking at the agent stff yeah I think speaking at a high level It's Our intention to keep a high level It's Our intention to keep a high level It's Our intention to keep investing a lot in you know making investing a lot in you know making investing a lot in you know making making the model better uh like I think making the model better uh like I think making the model better uh like I think I think uh you know we look at look at I think uh you know we look at look at I think uh you know we look at look at some of the you know some of the some of the you know some of the some of the you know some of the benchmarks where previous models were benchmarks where previous models were benchmarks where previous models were like oh could do it 6% of the time and like oh could do it 6% of the time and like oh could do it 6% of the time and now our model do at 14 or 22% of the now our model do at 14 or 22% of the now our model do at 14 or 22% of the time and yeah we want to get up to you time and yeah we want to get up to you time and yeah we want to get up to you know the human level reliability of 80 know the human level reliability of 80 know the human level reliability of 80 90% just like anywhere else right we're 90% just like anywhere else right we're 90% just like anywhere else right we're on the same curve that we were on with on the same curve that we were on with on the same curve that we were on with sbench where I think I would guess a sbench where I think I would guess a sbench where I think I would guess a year from now the models can do this year from now the models can do this year from now the models can do this very very reliably but you got to start

very very reliably but you got to start very very reliably but you got to start somewhere so you think it's possible to somewhere so you think it's possible to somewhere so you think it's possible to get to the the human level 90% uh get to the the human level 90% uh get to the the human level 90% uh basically doing the same thing you're basically doing the same thing you're basically doing the same thing you're doing now or is it has to be special for doing now or is it has to be special for doing now or is it has to be special for computer use I I mean uh depends what computer use I I mean uh depends what computer use I I mean uh depends what you mean by by you know special and FAL you mean by by you know special and FAL you mean by by you know special and FAL and and and um but but you know I generally think um but but you know I generally think um but but you know I generally think you know the same kinds of techniques you know the same kinds of techniques you know the same kinds of techniques that we've been using to train the that we've been using to train the that we've been using to train the current model I I expect that doubling current model I I expect that doubling current model I I expect that doubling down in those techniques in the same way down in those techniques in the same way down in those techniques in the same way that we have for code for code for that we have for code for code for that we have for code for code for models in general for other k for you models in general for other k for you models in general for other k for you know for image input um uh you know for know for image input um uh you know for know for image input um uh you know for voice uh I expect those same techniques voice uh I expect those same techniques voice uh I expect those same techniques will scale here as they have everywhere will scale here as they have everywhere will scale here as they have everywhere else but this is giving sort of the else but this is giving sort of the else but this is giving sort of the power of action to Claude And so you power of action to Claude And so you power of action to Claude And so you could do a lot of really powerful things could do a lot of really powerful things could do a lot of really powerful things but you could do a lot of damage also but you could do a lot of damage also but you could do a lot of damage also yeah yeah no and we've been very aware yeah yeah no and we've been very aware yeah yeah no and we've been very aware of that look my my view actually is of that look my my view actually is of that look my my view actually is computer use isn't a fundamentally new computer use isn't a fundamentally new computer use isn't a fundamentally new capability like the cbrn or autonomy capability like the cbrn or autonomy capability like the cbrn or autonomy capabilities are um it's more like it capabilities are um it's more like it capabilities are um it's more like it kind of opens the aperture for the model kind of opens the aperture for the model kind of opens the aperture for the model to use and apply its existing abilities to use and apply its existing abilities to use and apply its existing abilities uh and and so the way we think about it uh and and so the way we think about it uh and and so the way we think about it going back to our RSP is nothing that going back to our RSP is nothing that going back to our RSP is nothing that this model is this model is this model is doing inherently increases doing inherently increases doing inherently increases you know the risk from an RSP RSP you know the risk from an RSP RSP you know the risk from an RSP RSP perspective but as the models get more perspective but as the models get more perspective but as the models get more powerful having this capability may make powerful having this capability may make powerful having this capability may make it scarier once it you know once it has it scarier once it you know once it has it scarier once it you know once it has the cognitive capability to um you know the cognitive capability to um you know the cognitive capability to um you know to do something at the asl3 and asl4

to do something at the asl3 and asl4 to do something at the asl3 and asl4 level this this you know this may be the level this this you know this may be the level this this you know this may be the thing that kind of Unbound it from doing thing that kind of Unbound it from doing thing that kind of Unbound it from doing so so going forward certainly this so so going forward certainly this so so going forward certainly this modality of interaction is something we modality of interaction is something we modality of interaction is something we have tested for and that we will have tested for and that we will have tested for and that we will continue to test for in our going continue to test for in our going continue to test for in our going forward um I think it's probably better forward um I think it's probably better forward um I think it's probably better to have to learn and explore this to have to learn and explore this to have to learn and explore this capability before the model is super uh capability before the model is super uh capability before the model is super uh you know super capable yeah and there's you know super capable yeah and there's you know super capable yeah and there's uh a lot of interesting attacks like uh a lot of interesting attacks like uh a lot of interesting attacks like prompt injection because now you've prompt injection because now you've prompt injection because now you've widened the aperture so you can prompt widened the aperture so you can prompt widened the aperture so you can prompt inject through stuff on screen so if inject through stuff on screen so if inject through stuff on screen so if this becomes more and more useful then this becomes more and more useful then this becomes more and more useful then there's more and more benefit to inject there's more and more benefit to inject there's more and more benefit to inject inject stuff into the model if it goes inject stuff into the model if it goes inject stuff into the model if it goes to certain web page it could be harmless to certain web page it could be harmless to certain web page it could be harmless stuff like advertisements or it could be stuff like advertisements or it could be stuff like advertisements or it could be like harmful stuff right yeah I mean we like harmful stuff right yeah I mean we like harmful stuff right yeah I mean we thought a lot about like spam capture thought a lot about like spam capture thought a lot about like spam capture you know Mass C there's all you know you know Mass C there's all you know you know Mass C there's all you know every every like if one secret I'll tell every every like if one secret I'll tell every every like if one secret I'll tell you if you've invented a new technology you if you've invented a new technology you if you've invented a new technology not necessarily the biggest misuse but not necessarily the biggest misuse but not necessarily the biggest misuse but but the the first misuse you'll see but the the first misuse you'll see but the the first misuse you'll see scams just Petty scams like you just scams just Petty scams like you just scams just Petty scams like you just just just it's it's like it's like a just just it's it's like it's like a just just it's it's like it's like a thing as old people scamming each other thing as old people scamming each other thing as old people scamming each other it's it's this it's this thing as old as it's it's this it's this thing as old as it's it's this it's this thing as old as time um and and and it's just every time time um and and and it's just every time time um and and and it's just every time you got to deal with it it's almost like you got to deal with it it's almost like you got to deal with it it's almost like silly to say but it's it's true sort of silly to say but it's it's true sort of silly to say but it's it's true sort of and spam in general is a thing as it and spam in general is a thing as it and spam in general is a thing as it gets more and more intelligent it's uh gets more and more intelligent it's uh gets more and more intelligent it's uh there a lot of like like I said like there a lot of like like I said like there a lot of like like I said like there are a lot of petty criminals in there are a lot of petty criminals in there are a lot of petty criminals in the world and and and you know it's like the world and and and you know it's like the world and and and you know it's like every new technology is like a new way every new technology is like a new way every new technology is like a new way for petty petty criminals to do for petty petty criminals to do for petty petty criminals to do something you know something stupid and

something you know something stupid and something you know something stupid and malicious um is there any ideas about malicious um is there any ideas about malicious um is there any ideas about sandboxing it like how difficult is the sandboxing it like how difficult is the sandboxing it like how difficult is the sandboxing task yeah we sandbox during sandboxing task yeah we sandbox during sandboxing task yeah we sandbox during training so for example during training training so for example during training training so for example during training we didn't expose the model to the we didn't expose the model to the we didn't expose the model to the internet um I think that's probably a internet um I think that's probably a internet um I think that's probably a bad idea during training because uh you bad idea during training because uh you bad idea during training because uh you know the model can be changing its know the model can be changing its know the model can be changing its policy it can be changing what it's policy it can be changing what it's policy it can be changing what it's doing and it's having an effect in the doing and it's having an effect in the doing and it's having an effect in the real world um uh you know in in terms of real world um uh you know in in terms of real world um uh you know in in terms of actually deploying the model right it actually deploying the model right it actually deploying the model right it kind of depends on the application like kind of depends on the application like kind of depends on the application like you know sometimes you want the model to you know sometimes you want the model to you know sometimes you want the model to do something in the real world but of do something in the real world but of do something in the real world but of course you can always put guard you can course you can always put guard you can course you can always put guard you can always put guard rails on the outside always put guard rails on the outside always put guard rails on the outside right you can say okay well you know right you can say okay well you know right you can say okay well you know this model is not going to move data this model is not going to move data this model is not going to move data from my you know this model is not going from my you know this model is not going from my you know this model is not going to move any files from my computer to or to move any files from my computer to or to move any files from my computer to or my web server to anywhere else now when my web server to anywhere else now when my web server to anywhere else now when you talk about sandboxing again when we you talk about sandboxing again when we you talk about sandboxing again when we get to asl4 none of these precautions get to asl4 none of these precautions get to asl4 none of these precautions are going to make sense there right are going to make sense there right are going to make sense there right where when you when you talk about asl4 where when you when you talk about asl4 where when you when you talk about asl4 you're then the model is being kind of you're then the model is being kind of you're then the model is being kind of you know there's a a theoretical worry you know there's a a theoretical worry you know there's a a theoretical worry the model could be smart enough to break the model could be smart enough to break the model could be smart enough to break it to to kind of break out of any box it to to kind of break out of any box it to to kind of break out of any box and so there we need to think about and so there we need to think about and so there we need to think about mechanistic interpretability about you mechanistic interpretability about you mechanistic interpretability about you know if we're if we're going to have a know if we're if we're going to have a know if we're if we're going to have a Sandbox it would need to be a Sandbox it would need to be a Sandbox it would need to be a mathematically provable sound but you mathematically provable sound but you mathematically provable sound but you know that's that's a whole different know that's that's a whole different know that's that's a whole different world than what we're dealing with with world than what we're dealing with with world than what we're dealing with with the models the models the models today yeah the science of building a box today yeah the science of building a box today yeah the science of building a box from which asl4 AI system cannot Escape from which asl4 AI system cannot Escape from which asl4 AI system cannot Escape I I think it's probably not the right I I think it's probably not the right I I think it's probably not the right approach I think the right approach approach I think the right approach approach I think the right approach instead of having something you know

instead of having something you know instead of having something you know unaligned that that like you're trying unaligned that that like you're trying unaligned that that like you're trying to prevent it from escaping I think it's to prevent it from escaping I think it's to prevent it from escaping I think it's it's better to just design the model the it's better to just design the model the it's better to just design the model the right way or have a loop where you you right way or have a loop where you you right way or have a loop where you you know you look inside you look inside the know you look inside you look inside the know you look inside you look inside the model and you're able to verify property model and you're able to verify property model and you're able to verify property and that gives you an opportunity to and that gives you an opportunity to and that gives you an opportunity to like iterate and actually get it right like iterate and actually get it right like iterate and actually get it right um I think I think containing uh um I think I think containing uh um I think I think containing uh containing bad models is is is much containing bad models is is is much containing bad models is is is much worse solution than having good models worse solution than having good models worse solution than having good models let me ask about regulation what's the let me ask about regulation what's the let me ask about regulation what's the role of regulation in keeping AI safe so role of regulation in keeping AI safe so role of regulation in keeping AI safe so for example he described California AI for example he described California AI for example he described California AI regulation Bill SB 1047 that was regulation Bill SB 1047 that was regulation Bill SB 1047 that was ultimately vetoed by the governor what ultimately vetoed by the governor what ultimately vetoed by the governor what are the pros and cons of this bill are the pros and cons of this bill are the pros and cons of this bill General yes we ended up making some General yes we ended up making some General yes we ended up making some suggestions to the bill and then some of suggestions to the bill and then some of suggestions to the bill and then some of those were opted and you know we felt I those were opted and you know we felt I those were opted and you know we felt I think I think quite positively uh uh think I think quite positively uh uh think I think quite positively uh uh quite positively about about the bill uh quite positively about about the bill uh quite positively about about the bill uh by by the end of that um it did still by by the end of that um it did still by by the end of that um it did still have some downsides um uh and you know have some downsides um uh and you know have some downsides um uh and you know of course of course it got vetoed um I of course of course it got vetoed um I of course of course it got vetoed um I think at a high level I think some of think at a high level I think some of think at a high level I think some of the key ideas behind the bill um are you the key ideas behind the bill um are you the key ideas behind the bill um are you know I would say similar to ideas behind know I would say similar to ideas behind know I would say similar to ideas behind our rsps and I think it's very important our rsps and I think it's very important our rsps and I think it's very important that some jurisdiction whether it's that some jurisdiction whether it's that some jurisdiction whether it's California or the federal government California or the federal government California or the federal government Andor other countries and other states Andor other countries and other states Andor other countries and other states passes some regulation like this and I passes some regulation like this and I passes some regulation like this and I can talk through why I think that's so can talk through why I think that's so can talk through why I think that's so important so I feel good about our RSP important so I feel good about our RSP important so I feel good about our RSP it's not perfect it needs to be iterated it's not perfect it needs to be iterated it's not perfect it needs to be iterated on a lot but it's been a good forcing on a lot but it's been a good forcing on a lot but it's been a good forcing function for getting the company to take function for getting the company to take function for getting the company to take these risks seriously to put them into

these risks seriously to put them into these risks seriously to put them into product planning to really make them a product planning to really make them a product planning to really make them a central part of work at anthropic and to central part of work at anthropic and to central part of work at anthropic and to make sure that all the thousand people make sure that all the thousand people make sure that all the thousand people and it's almost a thousand people now at and it's almost a thousand people now at and it's almost a thousand people now at anthropic understand that this is one of anthropic understand that this is one of anthropic understand that this is one of the highest priorities of the company if the highest priorities of the company if the highest priorities of the company if not the highest priority uh not the highest priority uh not the highest priority uh but one there are some there are still but one there are some there are still but one there are some there are still some companies that don't have RSP like some companies that don't have RSP like some companies that don't have RSP like mechanisms like open aai Google uh did mechanisms like open aai Google uh did mechanisms like open aai Google uh did adopt these mechanisms a couple months adopt these mechanisms a couple months adopt these mechanisms a couple months after uh after anthropic did uh but after uh after anthropic did uh but after uh after anthropic did uh but there are there are other companies out there are there are other companies out there are there are other companies out there that don't have these mechanisms there that don't have these mechanisms there that don't have these mechanisms at all uh and so if some companies adopt at all uh and so if some companies adopt at all uh and so if some companies adopt these mechanisms and others don't uh these mechanisms and others don't uh these mechanisms and others don't uh it's really going to create a situation it's really going to create a situation it's really going to create a situation where you know some of these dangers where you know some of these dangers where you know some of these dangers have the property that it doesn't matter have the property that it doesn't matter have the property that it doesn't matter if three out of five of the companies if three out of five of the companies if three out of five of the companies are being safe if the other two are are are being safe if the other two are are are being safe if the other two are are being are being unsafe it creates this being are being unsafe it creates this being are being unsafe it creates this negative externality and and I think the negative externality and and I think the negative externality and and I think the lack of uniformity is not fair to those lack of uniformity is not fair to those lack of uniformity is not fair to those of us who have put a lot of effort into of us who have put a lot of effort into of us who have put a lot of effort into being very thoughtful about these being very thoughtful about these being very thoughtful about these procedures the second thing is I don't procedures the second thing is I don't procedures the second thing is I don't think you can trust these companies to think you can trust these companies to think you can trust these companies to adhere to these voluntary plans in their adhere to these voluntary plans in their adhere to these voluntary plans in their own right I like to think that anthropic own right I like to think that anthropic own right I like to think that anthropic will we do everything we can that we will we do everything we can that we will we do everything we can that we will our our our our RSP is checked by will our our our our RSP is checked by will our our our our RSP is checked by our long-term benefit trust uh so you our long-term benefit trust uh so you our long-term benefit trust uh so you know we do everything we can to to to know we do everything we can to to to know we do everything we can to to to adhere to our own RSP um but you know adhere to our own RSP um but you know adhere to our own RSP um but you know you hear lots of things about various you hear lots of things about various you hear lots of things about various companies saying oh they said they would companies saying oh they said they would companies saying oh they said they would do they said they would give this much do they said they would give this much do they said they would give this much compute and they didn't they said they

compute and they didn't they said they compute and they didn't they said they would do this thing and they didn't um would do this thing and they didn't um would do this thing and they didn't um you know I don't I don't think it makes you know I don't I don't think it makes you know I don't I don't think it makes sense to you know to to to you know sense to you know to to to you know sense to you know to to to you know litigate particular things that litigate particular things that litigate particular things that companies have done but I I think this companies have done but I I think this companies have done but I I think this this broad principle that like if this broad principle that like if this broad principle that like if there's nothing watching over them there's nothing watching over them there's nothing watching over them there's nothing watching over us as an there's nothing watching over us as an there's nothing watching over us as an industry there's no guarantee that we'll industry there's no guarantee that we'll industry there's no guarantee that we'll do the right thing and the stakes are do the right thing and the stakes are do the right thing and the stakes are very high uh and so I think it's I think very high uh and so I think it's I think very high uh and so I think it's I think it's important to have a uniform it's important to have a uniform it's important to have a uniform standard that that that that that standard that that that that that standard that that that that that everyone follows and to make sure that everyone follows and to make sure that everyone follows and to make sure that simply that the industry does what a simply that the industry does what a simply that the industry does what a majority of the industry has already majority of the industry has already majority of the industry has already said is important and has already said said is important and has already said said is important and has already said that they definitely will do right some that they definitely will do right some that they definitely will do right some people uh you know I think there's there people uh you know I think there's there people uh you know I think there's there a class of people who are against a class of people who are against a class of people who are against regulation on principle I understand regulation on principle I understand regulation on principle I understand where that comes from if you go to where that comes from if you go to where that comes from if you go to Europe and you know you see something Europe and you know you see something Europe and you know you see something like gdpr you see some of the other like gdpr you see some of the other like gdpr you see some of the other stuff that that that that that they've stuff that that that that that they've stuff that that that that that they've done you know some of it's good but but done you know some of it's good but but done you know some of it's good but but some of it is really unnecessarily some of it is really unnecessarily some of it is really unnecessarily burdensome and I think it's fair to say burdensome and I think it's fair to say burdensome and I think it's fair to say really has slowed really has slowed really has slowed really has slowed really has slowed really has slowed Innovation and so I understand where Innovation and so I understand where Innovation and so I understand where people are coming from on priors I people are coming from on priors I people are coming from on priors I understand why people come from start understand why people come from start understand why people come from start from that start from that position uh from that start from that position uh from that start from that position uh but but again I think AI is different if but but again I think AI is different if but but again I think AI is different if we go to the very serious risks of we go to the very serious risks of we go to the very serious risks of autonomy and misuse that that that I autonomy and misuse that that that I autonomy and misuse that that that I talked about you know just a just a few talked about you know just a just a few talked about you know just a just a few minutes ago I think that those are minutes ago I think that those are minutes ago I think that those are unusual and they weren't an unusually unusual and they weren't an unusually unusual and they weren't an unusually strong response uh and so I I think it's strong response uh and so I I think it's strong response uh and so I I think it's very important again um we need

very important again um we need very important again um we need something that everyone can get behind something that everyone can get behind something that everyone can get behind uh you know I think one of the issues uh you know I think one of the issues uh you know I think one of the issues with with with s1047 uh especially the original version s1047 uh especially the original version s1047 uh especially the original version of it was it it had a bunch of the of it was it it had a bunch of the of it was it it had a bunch of the structure of rsps but it also had a structure of rsps but it also had a structure of rsps but it also had a bunch of stuff that was either clunky or bunch of stuff that was either clunky or bunch of stuff that was either clunky or that that that just would have created a that that that just would have created a that that that just would have created a bunch of burdens a bunch of Hassle and bunch of burdens a bunch of Hassle and bunch of burdens a bunch of Hassle and might even have missed the Target in might even have missed the Target in might even have missed the Target in terms of addressing the risks um you terms of addressing the risks um you terms of addressing the risks um you don't really hear about it on Twitter don't really hear about it on Twitter don't really hear about it on Twitter you just hear about kind of you know you just hear about kind of you know you just hear about kind of you know people are people are cheering for any people are people are cheering for any people are people are cheering for any regulation and then the folks who are regulation and then the folks who are regulation and then the folks who are against make up these often quite against make up these often quite against make up these often quite intellectually dishonest arguments about intellectually dishonest arguments about intellectually dishonest arguments about how you know it you know it'll make us how you know it you know it'll make us how you know it you know it'll make us move away from California bill bill move away from California bill bill move away from California bill bill doesn't apply if you're headquartered in doesn't apply if you're headquartered in doesn't apply if you're headquartered in California bill only applies if you do California bill only applies if you do California bill only applies if you do business in California um or that it business in California um or that it business in California um or that it would damage the open source ecosystem would damage the open source ecosystem would damage the open source ecosystem or that it would you know it would cause or that it would you know it would cause or that it would you know it would cause cause all of these things I I think cause all of these things I I think cause all of these things I I think those were mostly nonsense but there are those were mostly nonsense but there are those were mostly nonsense but there are better arguments against regulation better arguments against regulation better arguments against regulation there's one guy uh Dean ball who's there's one guy uh Dean ball who's there's one guy uh Dean ball who's really you know I think a very scholarly really you know I think a very scholarly really you know I think a very scholarly scholarly IST who who looks at what scholarly IST who who looks at what scholarly IST who who looks at what happens when a regulation is put in happens when a regulation is put in happens when a regulation is put in place and ways that they can kind of get place and ways that they can kind of get place and ways that they can kind of get a life of their own or how they can be a life of their own or how they can be a life of their own or how they can be poorly designed and so our interest has poorly designed and so our interest has poorly designed and so our interest has always been we do think there should be always been we do think there should be always been we do think there should be regulation in this space but we want to regulation in this space but we want to regulation in this space but we want to be an actor who makes sure that that be an actor who makes sure that that be an actor who makes sure that that that that regulation is something that's that that regulation is something that's that that regulation is something that's surgical that's targeted at the serious

surgical that's targeted at the serious surgical that's targeted at the serious risks and is something people can risks and is something people can risks and is something people can actually comply with because something I actually comply with because something I actually comply with because something I think The Advocates of Regulation don't think The Advocates of Regulation don't think The Advocates of Regulation don't understand as well as they could is if understand as well as they could is if understand as well as they could is if we get something in place that is um we get something in place that is um we get something in place that is um that's poorly targeted that wastes a that's poorly targeted that wastes a that's poorly targeted that wastes a bunch of people's time what's going to bunch of people's time what's going to bunch of people's time what's going to happen is people are going to say see happen is people are going to say see happen is people are going to say see these safety risks there you know this these safety risks there you know this these safety risks there you know this is this is nonsense I just you know I is this is nonsense I just you know I is this is nonsense I just you know I just had to hire 10 lawyers to to you just had to hire 10 lawyers to to you just had to hire 10 lawyers to to you know to fill out all these forms I had know to fill out all these forms I had know to fill out all these forms I had to run all these tests for something to run all these tests for something to run all these tests for something that was clearly not dangerous and after that was clearly not dangerous and after that was clearly not dangerous and after 6 months of that there will be there 6 months of that there will be there 6 months of that there will be there will be a ground sweep well and we'll will be a ground sweep well and we'll will be a ground sweep well and we'll we'll we'll we'll end up with a durable we'll we'll we'll end up with a durable we'll we'll we'll end up with a durable consensus against regulation and so the consensus against regulation and so the consensus against regulation and so the I I think the the worst enemy of those I I think the the worst enemy of those I I think the the worst enemy of those who want real accountability is badly who want real accountability is badly who want real accountability is badly designed regulation um we we need to designed regulation um we we need to designed regulation um we we need to actually get it right uh and and this is actually get it right uh and and this is actually get it right uh and and this is if there's one thing I could say to The if there's one thing I could say to The if there's one thing I could say to The Advocates it it would be that I want Advocates it it would be that I want Advocates it it would be that I want them to understand this Dynamic better them to understand this Dynamic better them to understand this Dynamic better and we need to be really careful and we and we need to be really careful and we and we need to be really careful and we need to talk to people who actually have need to talk to people who actually have need to talk to people who actually have who actually have experience seeing how who actually have experience seeing how who actually have experience seeing how regulations play out in practice and and regulations play out in practice and and regulations play out in practice and and the people who have seen that understand the people who have seen that understand the people who have seen that understand to be very careful if this was some to be very careful if this was some to be very careful if this was some lesser issue I might be against lesser issue I might be against lesser issue I might be against regulation at all but what what I want regulation at all but what what I want regulation at all but what what I want the opponents to understand is is that the opponents to understand is is that the opponents to understand is is that the underlying issues are actually the underlying issues are actually the underlying issues are actually serious they're they're not they're not serious they're they're not they're not serious they're they're not they're not something that I or the other companies something that I or the other companies something that I or the other companies are just making up because of regulatory are just making up because of regulatory are just making up because of regulatory capture they're not sci-fi fantasies

capture they're not sci-fi fantasies capture they're not sci-fi fantasies they're not they're not any of these they're not they're not any of these they're not they're not any of these things um you know every every time we things um you know every every time we things um you know every every time we have new model every few months we have new model every few months we have new model every few months we measure the behavior of these models and measure the behavior of these models and measure the behavior of these models and they're getting better and better at they're getting better and better at they're getting better and better at these concerning tasks just as they are these concerning tasks just as they are these concerning tasks just as they are getting better and better at um you know getting better and better at um you know getting better and better at um you know good valuable economically useful tasks good valuable economically useful tasks good valuable economically useful tasks and so I I I I would just love it if and so I I I I would just love it if and so I I I I would just love it if some of the former you know I think some of the former you know I think some of the former you know I think sb147 was very polarizing I would love sb147 was very polarizing I would love sb147 was very polarizing I would love it if some of the most reasonable it if some of the most reasonable it if some of the most reasonable opponents and some of the most opponents and some of the most opponents and some of the most reasonable um uh proponents uh would sit reasonable um uh proponents uh would sit reasonable um uh proponents uh would sit down together and you know I think I down together and you know I think I down together and you know I think I think that you know the different the think that you know the different the think that you know the different the different AI companies um you know different AI companies um you know different AI companies um you know anthropic was the the only AI company anthropic was the the only AI company anthropic was the the only AI company that you know felt positively in a very that you know felt positively in a very that you know felt positively in a very detailed way I think Elon tweeted uh detailed way I think Elon tweeted uh detailed way I think Elon tweeted uh tweeted briefly something positive but tweeted briefly something positive but tweeted briefly something positive but you know some of the some of the big you know some of the some of the big you know some of the some of the big ones like Google open AI meta Microsoft ones like Google open AI meta Microsoft ones like Google open AI meta Microsoft were were pretty St stly against so I were were pretty St stly against so I were were pretty St stly against so I would really like is if if you know some would really like is if if you know some would really like is if if you know some of the key stakeholders some of the you of the key stakeholders some of the you of the key stakeholders some of the you know thoughtful proponents and and some know thoughtful proponents and and some know thoughtful proponents and and some of the most thoughtful opponents would of the most thoughtful opponents would of the most thoughtful opponents would sit down and say how do we solve this sit down and say how do we solve this sit down and say how do we solve this problem in in a way that the proponents problem in in a way that the proponents problem in in a way that the proponents feel brings a real reduction in risk and feel brings a real reduction in risk and feel brings a real reduction in risk and that the opponents feel that it is not that the opponents feel that it is not that the opponents feel that it is not it is not hampering the the industry or it is not hampering the the industry or it is not hampering the the industry or hampering Innovation any more necessary hampering Innovation any more necessary hampering Innovation any more necessary than it than than than it needs to and than it than than than it needs to and than it than than than it needs to and and I think for for whatever reason that

and I think for for whatever reason that and I think for for whatever reason that things got too polarized and those two things got too polarized and those two things got too polarized and those two groups groups groups didn't get to sit down in the way that didn't get to sit down in the way that didn't get to sit down in the way that they should uh and and I feel I feel they should uh and and I feel I feel they should uh and and I feel I feel urgency I really think we need to do urgency I really think we need to do urgency I really think we need to do something in something in something in 2025 uh uh you know if we get to the end 2025 uh uh you know if we get to the end 2025 uh uh you know if we get to the end of 2025 and we've still done nothing of 2025 and we've still done nothing of 2025 and we've still done nothing about this then I'm going to be worried about this then I'm going to be worried about this then I'm going to be worried I'm not I'm not worried yet because I'm not I'm not worried yet because I'm not I'm not worried yet because again the risks aren't here yet but but again the risks aren't here yet but but again the risks aren't here yet but but I I I think time is running short yeah I I I think time is running short yeah I I I think time is running short yeah and come up with something surgical like and come up with something surgical like and come up with something surgical like you said yeah yeah yeah exactly and and you said yeah yeah yeah exactly and and you said yeah yeah yeah exactly and and we need to get we need to get away from we need to get we need to get away from we need to get we need to get away from this this this intense this this this intense this this this intense pro- safety versus intense pro- safety versus intense pro- safety versus intense anti-regulatory rhetoric right it's anti-regulatory rhetoric right it's anti-regulatory rhetoric right it's turned into these these flame Wars on turned into these these flame Wars on turned into these these flame Wars on Twitter and nothing Good's going to come Twitter and nothing Good's going to come Twitter and nothing Good's going to come with that so there's a lot of curiosity with that so there's a lot of curiosity with that so there's a lot of curiosity about the different players in the game about the different players in the game about the different players in the game one of the uh ogs is open AI you have one of the uh ogs is open AI you have one of the uh ogs is open AI you have had several years of experience at open had several years of experience at open had several years of experience at open AI what's your story and history there AI what's your story and history there AI what's your story and history there yeah so I was at open AI for uh for yeah so I was at open AI for uh for yeah so I was at open AI for uh for roughly five years uh for the last I roughly five years uh for the last I roughly five years uh for the last I think it was a couple years you know I I think it was a couple years you know I I think it was a couple years you know I I I I I I was uh vice president of I I I I was uh vice president of I I I I was uh vice president of research there um probably myself and research there um probably myself and research there um probably myself and Ilia suger were the ones who you know Ilia suger were the ones who you know Ilia suger were the ones who you know really kind of set the set the research really kind of set the set the research really kind of set the set the research Direction around 2016 or 2017 I first Direction around 2016 or 2017 I first Direction around 2016 or 2017 I first started to really believe in or at least started to really believe in or at least started to really believe in or at least confirm my belief in the scaling confirm my belief in the scaling confirm my belief in the scaling hypothesis when when Ilia famously said hypothesis when when Ilia famously said hypothesis when when Ilia famously said to me the thing you need to understand to me the thing you need to understand to me the thing you need to understand about these models is they just want to about these models is they just want to about these models is they just want to learn the models just want to learn um learn the models just want to learn um learn the models just want to learn um and and and and again sometimes there and and and and again sometimes there and and and and again sometimes there are these One S there these one are these One S there these one are these One S there these one sentences these Zen cones that you hear sentences these Zen cones that you hear sentences these Zen cones that you hear them and you're like ah that that

them and you're like ah that that them and you're like ah that that explains everything that explains like a explains everything that explains like a explains everything that explains like a thousand things that I've seen and then thousand things that I've seen and then thousand things that I've seen and then and then I I you know ever after I had and then I I you know ever after I had and then I I you know ever after I had this visualization in my head of like this visualization in my head of like this visualization in my head of like you optimize the models in the right way you optimize the models in the right way you optimize the models in the right way you point the models in the right way you point the models in the right way you point the models in the right way they just want to learn they just want they just want to learn they just want they just want to learn they just want to solve the problem regardless of what to solve the problem regardless of what to solve the problem regardless of what the problem is so get out of their way the problem is so get out of their way the problem is so get out of their way basically get out of their way yeah basically get out of their way yeah basically get out of their way yeah don't impose your own ideas about how don't impose your own ideas about how don't impose your own ideas about how they should learn and you know this was they should learn and you know this was they should learn and you know this was the same thing as Rich Sutton put out in the same thing as Rich Sutton put out in the same thing as Rich Sutton put out in the bitter lesson or G put out in the the bitter lesson or G put out in the the bitter lesson or G put out in the scaling hypothesis you know I think scaling hypothesis you know I think scaling hypothesis you know I think generally the dynamic was you know I got generally the dynamic was you know I got generally the dynamic was you know I got I got this kind of inspiration from uh I got this kind of inspiration from uh I got this kind of inspiration from uh from from from Ilan from others folks from from from Ilan from others folks from from from Ilan from others folks like Alec Radford who did the the like Alec Radford who did the the like Alec Radford who did the the original uh uh original uh uh original uh uh gpt1 uh and then uh ran really hard with gpt1 uh and then uh ran really hard with gpt1 uh and then uh ran really hard with it me me and my collaborators on gpt2 it me me and my collaborators on gpt2 it me me and my collaborators on gpt2 gpt3 RL from Human feedback which was an gpt3 RL from Human feedback which was an gpt3 RL from Human feedback which was an attempt to kind of deal with the early attempt to kind of deal with the early attempt to kind of deal with the early safety and durability things like debate safety and durability things like debate safety and durability things like debate and amplification heavy on and amplification heavy on and amplification heavy on interpretability so again the interpretability so again the interpretability so again the combination of safety plus scaling combination of safety plus scaling combination of safety plus scaling probably 2018 2019 2020 those those were probably 2018 2019 2020 those those were probably 2018 2019 2020 those those were those were kind of the years when myself those were kind of the years when myself those were kind of the years when myself and my collaborators probably um you and my collaborators probably um you and my collaborators probably um you know mo mo many many of whom became know mo mo many many of whom became know mo mo many many of whom became co-founders of anthropic kind of really co-founders of anthropic kind of really co-founders of anthropic kind of really had had had a vision and like and like had had had a vision and like and like had had had a vision and like and like drove the direction why'd you leave why' drove the direction why'd you leave why' drove the direction why'd you leave why' you decid to leave yeah so look I'm you decid to leave yeah so look I'm you decid to leave yeah so look I'm gonna put things this way and I you know gonna put things this way and I you know gonna put things this way and I you know I think it I think it ties to the to to I think it I think it ties to the to to I think it I think it ties to the to to the race to the top right which is you the race to the top right which is you the race to the top right which is you know in my time at open AI what I come

know in my time at open AI what I come know in my time at open AI what I come come to see as I'd come to appreciate come to see as I'd come to appreciate come to see as I'd come to appreciate the scaling hypothesis and as I'd come the scaling hypothesis and as I'd come the scaling hypothesis and as I'd come to appreciate kind of the importance of to appreciate kind of the importance of to appreciate kind of the importance of safety along with the scaling hypothesis safety along with the scaling hypothesis safety along with the scaling hypothesis the first one I think you know open AI the first one I think you know open AI the first one I think you know open AI was was getting was getting on board was was getting was getting on board was was getting was getting on board with um the second one in a way had with um the second one in a way had with um the second one in a way had always been part of of open ai's always been part of of open ai's always been part of of open ai's messaging um but uh you know over over messaging um but uh you know over over messaging um but uh you know over over many years of of the time the time that many years of of the time the time that many years of of the time the time that I spent there I think I had a particular I spent there I think I had a particular I spent there I think I had a particular vision of how these how we should handle vision of how these how we should handle vision of how these how we should handle these things how we should be brought these things how we should be brought these things how we should be brought out in the world the kind of principles out in the world the kind of principles out in the world the kind of principles that the organization should have and that the organization should have and that the organization should have and look I mean there were like many many look I mean there were like many many look I mean there were like many many discussions about like you know should discussions about like you know should discussions about like you know should the or do should the company do this the or do should the company do this the or do should the company do this should the company do that like there's should the company do that like there's should the company do that like there's a bunch of misinformation out there a bunch of misinformation out there a bunch of misinformation out there people say like we left because we people say like we left because we people say like we left because we didn't like the deal with Microsoft didn't like the deal with Microsoft didn't like the deal with Microsoft false although you know there was like a false although you know there was like a false although you know there was like a lot of discussion a lot of questions lot of discussion a lot of questions lot of discussion a lot of questions about exactly how we do the deal with about exactly how we do the deal with about exactly how we do the deal with Microsoft um we left because we didn't Microsoft um we left because we didn't Microsoft um we left because we didn't like commercialization that's not true like commercialization that's not true like commercialization that's not true we built gbd3 which was the model that we built gbd3 which was the model that we built gbd3 which was the model that was commercialized I was involved in was commercialized I was involved in was commercialized I was involved in commercialization it's it's more again commercialization it's it's more again commercialization it's it's more again about how do you do it like Civilization about how do you do it like Civilization about how do you do it like Civilization is going down this path to very powerful is going down this path to very powerful is going down this path to very powerful AI what's the way to do it that is AI what's the way to do it that is AI what's the way to do it that is cautious cautious cautious straightforward honest um that build straightforward honest um that build straightforward honest um that build trust in the organization and in trust in the organization and in trust in the organization and in individuals how do we get from here to individuals how do we get from here to individuals how do we get from here to there and how do we have a real vision there and how do we have a real vision there and how do we have a real vision for how to get it right how can safety for how to get it right how can safety for how to get it right how can safety not just be something we say because it not just be something we say because it not just be something we say because it helps with recruiting um and you know I

helps with recruiting um and you know I helps with recruiting um and you know I think I think at the end of the day um think I think at the end of the day um think I think at the end of the day um if you have a vision for that forget if you have a vision for that forget if you have a vision for that forget about anyone else's Vision I don't want about anyone else's Vision I don't want about anyone else's Vision I don't want to talk about anyone else's Vision if to talk about anyone else's Vision if to talk about anyone else's Vision if you have a vision for how to do it you you have a vision for how to do it you you have a vision for how to do it you should go off and you should do that should go off and you should do that should go off and you should do that Vision it is incredibly unproductive to Vision it is incredibly unproductive to Vision it is incredibly unproductive to try and argue with someone else's Vision try and argue with someone else's Vision try and argue with someone else's Vision you might think they're not doing it the you might think they're not doing it the you might think they're not doing it the right way you might think they're right way you might think they're right way you might think they're they're they're dishonest who knows they're they're dishonest who knows they're they're dishonest who knows maybe you're right maybe you're not um maybe you're right maybe you're not um maybe you're right maybe you're not um uh but uh what what you should do is you uh but uh what what you should do is you uh but uh what what you should do is you should take some people you trust and should take some people you trust and should take some people you trust and you should go off together and you you should go off together and you you should go off together and you should make your vision happen and if should make your vision happen and if should make your vision happen and if your vision is compelling if you can your vision is compelling if you can your vision is compelling if you can make it appeal to people some you know make it appeal to people some you know make it appeal to people some you know some combination of ethically you know some combination of ethically you know some combination of ethically you know in the market uh you know if if you can in the market uh you know if if you can in the market uh you know if if you can if you can make a company that's a place if you can make a company that's a place if you can make a company that's a place people want to join uh that you know people want to join uh that you know people want to join uh that you know engages in practices that people think engages in practices that people think engages in practices that people think are are reasonable while managing to are are reasonable while managing to are are reasonable while managing to maintain its position in the ecosystem maintain its position in the ecosystem maintain its position in the ecosystem at the same time if you do that people at the same time if you do that people at the same time if you do that people will copy it um and the fact that you will copy it um and the fact that you will copy it um and the fact that you were doing it especially the fact that were doing it especially the fact that were doing it especially the fact that you're doing it better than they are um you're doing it better than they are um you're doing it better than they are um causes them to change their behavior in causes them to change their behavior in causes them to change their behavior in a much more compelling way than if a much more compelling way than if a much more compelling way than if they're your boss and you're arguing they're your boss and you're arguing they're your boss and you're arguing with them I just I don't know how to be with them I just I don't know how to be with them I just I don't know how to be any more specific about it than that but any more specific about it than that but any more specific about it than that but I think it's generally very unproductive I think it's generally very unproductive I think it's generally very unproductive to try and get someone else's Vision to to try and get someone else's Vision to to try and get someone else's Vision to look like your vision um it's much more look like your vision um it's much more look like your vision um it's much more productive to go off and do a clean productive to go off and do a clean productive to go off and do a clean experiment and say this is our vision experiment and say this is our vision experiment and say this is our vision this is how this is this is how we're this is how this is this is how we're this is how this is this is how we're going to do things your choice is you going to do things your choice is you going to do things your choice is you can you can ignore us you can reject can you can ignore us you can reject can you can ignore us you can reject what we're doing or you can you can

what we're doing or you can you can what we're doing or you can you can start to become more like us and start to become more like us and start to become more like us and imitation is the sincerest form of imitation is the sincerest form of imitation is the sincerest form of flattery um and you know that that that flattery um and you know that that that flattery um and you know that that that plays out in the behavior of customers plays out in the behavior of customers plays out in the behavior of customers that PS out in the behavior of the that PS out in the behavior of the that PS out in the behavior of the public that plays out in the behavior of public that plays out in the behavior of public that plays out in the behavior of where people choose to work uh and again where people choose to work uh and again where people choose to work uh and again again at the end it's it's not about one again at the end it's it's not about one again at the end it's it's not about one company winning or another company company winning or another company company winning or another company winning if if we or another company are winning if if we or another company are winning if if we or another company are engaging in some practice that you know engaging in some practice that you know engaging in some practice that you know people people find genuinely appealing people people find genuinely appealing people people find genuinely appealing and I want it to be in substance not and I want it to be in substance not and I want it to be in substance not just not just in appearance um and you just not just in appearance um and you just not just in appearance um and you know I think I think researchers are know I think I think researchers are know I think I think researchers are sophisticated and they look at substance sophisticated and they look at substance sophisticated and they look at substance uh and then other companies start uh and then other companies start uh and then other companies start copying that practice and they win copying that practice and they win copying that practice and they win because they copied that practice that's because they copied that practice that's because they copied that practice that's great that's success that's like the great that's success that's like the great that's success that's like the race to the top it doesn't matter who race to the top it doesn't matter who race to the top it doesn't matter who wins in the end as long as everyone is wins in the end as long as everyone is wins in the end as long as everyone is copying everyone else's good practices copying everyone else's good practices copying everyone else's good practices right one way I think of it is like the right one way I think of it is like the right one way I think of it is like the thing we're all afraid of is a race the thing we're all afraid of is a race the thing we're all afraid of is a race the bottom right in the race to the bottom bottom right in the race to the bottom bottom right in the race to the bottom doesn't matter who wins because we all doesn't matter who wins because we all doesn't matter who wins because we all lose right like you know in the most lose right like you know in the most lose right like you know in the most extreme world we we make this autonomous extreme world we we make this autonomous extreme world we we make this autonomous AI that you know the robots enslave us AI that you know the robots enslave us AI that you know the robots enslave us or whatever right I mean that's half or whatever right I mean that's half or whatever right I mean that's half joking but you know that that is the joking but you know that that is the joking but you know that that is the most extreme uh uh thing thing that most extreme uh uh thing thing that most extreme uh uh thing thing that could happen then then it doesn't matter could happen then then it doesn't matter could happen then then it doesn't matter which company was ahead um if instead which company was ahead um if instead which company was ahead um if instead you create a race to the top where you create a race to the top where you create a race to the top where people are competing to engage in good people are competing to engage in good people are competing to engage in good in good practices uh then you know at at in good practices uh then you know at at in good practices uh then you know at at the end of the day you know it doesn't the end of the day you know it doesn't the end of the day you know it doesn't matter who who ends up who ends up matter who who ends up who ends up matter who who ends up who ends up winning doesn't even matter who who winning doesn't even matter who who winning doesn't even matter who who started the race to the top the point started the race to the top the point started the race to the top the point isn't to be virtuous the point is to get

isn't to be virtuous the point is to get isn't to be virtuous the point is to get the system into a better equilibrium the system into a better equilibrium the system into a better equilibrium than it was before and and individual than it was before and and individual than it was before and and individual companies can play some role in doing companies can play some role in doing companies can play some role in doing this individual companies can can you this individual companies can can you this individual companies can can you know can help to start it can help to know can help to start it can help to know can help to start it can help to accelerate it and frankly I think accelerate it and frankly I think accelerate it and frankly I think individuals at other companies have have individuals at other companies have have individuals at other companies have have done this as well right the individuals done this as well right the individuals done this as well right the individuals that when we put out an RSP react by that when we put out an RSP react by that when we put out an RSP react by pushing harder to to to get something pushing harder to to to get something pushing harder to to to get something similar done get something similar done similar done get something similar done similar done get something similar done at at at other companies sometimes other at at at other companies sometimes other at at at other companies sometimes other companies do something that's like we're companies do something that's like we're companies do something that's like we're like oh it's a good practice we think we like oh it's a good practice we think we like oh it's a good practice we think we think that's good we should adopt it too think that's good we should adopt it too think that's good we should adopt it too the only difference is you know I think the only difference is you know I think the only difference is you know I think I think we are um we try to be more I think we are um we try to be more I think we are um we try to be more forward leaning we try and adopt more of forward leaning we try and adopt more of forward leaning we try and adopt more of these practices first and adopt them these practices first and adopt them these practices first and adopt them more quickly when others when others more quickly when others when others more quickly when others when others invent them but I think this Dynamic is invent them but I think this Dynamic is invent them but I think this Dynamic is what we should be pointing at and that I what we should be pointing at and that I what we should be pointing at and that I think I think it abstracts away the think I think it abstracts away the think I think it abstracts away the question of you know which company's question of you know which company's question of you know which company's winning who trusts who I I think all winning who trusts who I I think all winning who trusts who I I think all these all these questions of drama are these all these questions of drama are these all these questions of drama are are profoundly uninteresting and and the are profoundly uninteresting and and the are profoundly uninteresting and and the the thing that matters is the ecosystem the thing that matters is the ecosystem the thing that matters is the ecosystem that we all operate in and how to make that we all operate in and how to make that we all operate in and how to make that ecosystem better because that that ecosystem better because that that ecosystem better because that constrains all the players and so constrains all the players and so constrains all the players and so anthropic is this kind of clean anthropic is this kind of clean anthropic is this kind of clean experiment built on a foundation of like experiment built on a foundation of like experiment built on a foundation of like what concretely AI safety should look what concretely AI safety should look what concretely AI safety should look like we look I'm sure we've made plenty like we look I'm sure we've made plenty like we look I'm sure we've made plenty of mistakes along the way the perfect of mistakes along the way the perfect of mistakes along the way the perfect organization doesn't exist it has to organization doesn't exist it has to organization doesn't exist it has to deal with the the imperfection of a deal with the the imperfection of a deal with the the imperfection of a thousand employees it has to deal deal thousand employees it has to deal deal thousand employees it has to deal deal with the imperfection of our leaders with the imperfection of our leaders with the imperfection of our leaders including me it has to deal with the including me it has to deal with the including me it has to deal with the imperfection of the people we've put imperfection of the people we've put imperfection of the people we've put we've put to you know to oversee the

we've put to you know to oversee the we've put to you know to oversee the imperfection of the of the leaders like imperfection of the of the leaders like imperfection of the of the leaders like the like the board and the long-term the like the board and the long-term the like the board and the long-term benefit trust it's it's all it's all a benefit trust it's it's all it's all a benefit trust it's it's all it's all a set of imperfect people trying to aim set of imperfect people trying to aim set of imperfect people trying to aim imperfectly at some ideal that will imperfectly at some ideal that will imperfectly at some ideal that will never perfectly be achieved um that's never perfectly be achieved um that's never perfectly be achieved um that's what you sign up for that's what it will what you sign up for that's what it will what you sign up for that's what it will always be but uh uh imperfect doesn't always be but uh uh imperfect doesn't always be but uh uh imperfect doesn't mean you just give up there's better and mean you just give up there's better and mean you just give up there's better and there's worse and hopefully hopefully we there's worse and hopefully hopefully we there's worse and hopefully hopefully we can begin to build we can do well enough can begin to build we can do well enough can begin to build we can do well enough that we can begin to build some that we can begin to build some that we can begin to build some practices that the whole industry practices that the whole industry practices that the whole industry engages in and then you know my guess is engages in and then you know my guess is engages in and then you know my guess is that M multiple of these companies will that M multiple of these companies will that M multiple of these companies will be successful anthropic will be be successful anthropic will be be successful anthropic will be successful these other companies like successful these other companies like successful these other companies like ones I've been at the past will also be ones I've been at the past will also be ones I've been at the past will also be successful and some will be more successful and some will be more successful and some will be more successful than others that's less successful than others that's less successful than others that's less important than again that we we align important than again that we we align important than again that we we align the incentives of the industry and that the incentives of the industry and that the incentives of the industry and that happens partly through the race to the happens partly through the race to the happens partly through the race to the top partly through things like RSP top partly through things like RSP top partly through things like RSP partly through again selected surgical partly through again selected surgical partly through again selected surgical regulation you said Talent density beats regulation you said Talent density beats regulation you said Talent density beats Talent Talent Talent Mass so can you explain that can you Mass so can you explain that can you Mass so can you explain that can you expand on it can you just talk about expand on it can you just talk about expand on it can you just talk about what it takes to build a great team of what it takes to build a great team of what it takes to build a great team of AI researchers and Engineers this is one AI researchers and Engineers this is one AI researchers and Engineers this is one of these statements that's like more of these statements that's like more of these statements that's like more true every every every month every month true every every every month every month true every every every month every month I see this statement as more true than I I see this statement as more true than I I see this statement as more true than I did the month before so if I were to do did the month before so if I were to do did the month before so if I were to do a thought experiment let's say you have a thought experiment let's say you have a thought experiment let's say you have a team of 100 people that are super a team of 100 people that are super a team of 100 people that are super smart motivated and aligned with the smart motivated and aligned with the smart motivated and aligned with the mission and that's your company or you mission and that's your company or you mission and that's your company or you can have a team of a thousand people can have a team of a thousand people can have a team of a thousand people where 200 people are super smart super where 200 people are super smart super where 200 people are super smart super aligned with the mission and then uh

aligned with the mission and then uh aligned with the mission and then uh like and then like 800 people are let's like and then like 800 people are let's like and then like 800 people are let's just say you pick 800 like random random just say you pick 800 like random random just say you pick 800 like random random big Tech employees which would you big Tech employees which would you big Tech employees which would you rather have right the talent mass is rather have right the talent mass is rather have right the talent mass is greater in in the group of uh in the greater in in the group of uh in the greater in in the group of uh in the group of a thousand people right you group of a thousand people right you group of a thousand people right you have you have even even a larger number have you have even even a larger number have you have even even a larger number of incredibly talented incredibly of incredibly talented incredibly of incredibly talented incredibly aligned incredibly smart people um uh aligned incredibly smart people um uh aligned incredibly smart people um uh but but the the issue is just that but but the the issue is just that but but the the issue is just that if every time someone super talented if every time someone super talented if every time someone super talented looks around they see someone else super looks around they see someone else super looks around they see someone else super talented and super dedicated that sets talented and super dedicated that sets talented and super dedicated that sets the tone for everything right that sets the tone for everything right that sets the tone for everything right that sets the tone for everyone is super inspired the tone for everyone is super inspired the tone for everyone is super inspired to work at the same place everyone to work at the same place everyone to work at the same place everyone trusts everyone else if you have a trusts everyone else if you have a trusts everyone else if you have a thousand or 10,000 uh people and and thousand or 10,000 uh people and and thousand or 10,000 uh people and and things have really regressed right you things have really regressed right you things have really regressed right you are not able to do selection and you're are not able to do selection and you're are not able to do selection and you're choosing random people what happens is choosing random people what happens is choosing random people what happens is then you need to put a lot of proc CES then you need to put a lot of proc CES then you need to put a lot of proc CES and a lot of guard rails in place um and a lot of guard rails in place um and a lot of guard rails in place um just because people don't fully trust just because people don't fully trust just because people don't fully trust each other you have to adjudicate each other you have to adjudicate each other you have to adjudicate political battles like there are so many political battles like there are so many political battles like there are so many things that slow down the org's ability things that slow down the org's ability things that slow down the org's ability to operate and so we're nearly a to operate and so we're nearly a to operate and so we're nearly a thousand people and you know we've we've thousand people and you know we've we've thousand people and you know we've we've we've tried to make it so that as large we've tried to make it so that as large we've tried to make it so that as large a fraction of those thousand people as a fraction of those thousand people as a fraction of those thousand people as possible are like super talented super possible are like super talented super possible are like super talented super skilled it's one of the reasons we've skilled it's one of the reasons we've skilled it's one of the reasons we've we've slowed down hiring a lot in the we've slowed down hiring a lot in the we've slowed down hiring a lot in the last few months We Grew From 300 to 800 last few months We Grew From 300 to 800 last few months We Grew From 300 to 800 I believe I think in the first seven I believe I think in the first seven I believe I think in the first seven eight months of the year and now we've eight months of the year and now we've eight months of the year and now we've slowed down we're at like you know last slowed down we're at like you know last slowed down we're at like you know last three months we went from 800 to 900 950

three months we went from 800 to 900 950 three months we went from 800 to 900 950 something like that don't quote me on something like that don't quote me on something like that don't quote me on the exact numbers but I think there's an the exact numbers but I think there's an the exact numbers but I think there's an inflection point around a thousand and inflection point around a thousand and inflection point around a thousand and we want to be much more careful how how we want to be much more careful how how we want to be much more careful how how we how we grow uh early on and and now we how we grow uh early on and and now we how we grow uh early on and and now as well you know we've hired a lot of as well you know we've hired a lot of as well you know we've hired a lot of physicists um you know theoretical physicists um you know theoretical physicists um you know theoretical physicists can learn things really fast physicists can learn things really fast physicists can learn things really fast um uh even even more recently as we've um uh even even more recently as we've um uh even even more recently as we've continued to hire that you know we've continued to hire that you know we've continued to hire that you know we've really had a high bar for on both the really had a high bar for on both the really had a high bar for on both the research side and the software research side and the software research side and the software engineering side have hired a lot of engineering side have hired a lot of engineering side have hired a lot of senior people including folks who used senior people including folks who used senior people including folks who used to be at other at other companies in to be at other at other companies in to be at other at other companies in this space and we we've just continued this space and we we've just continued this space and we we've just continued to be very selective it's very easy to to be very selective it's very easy to to be very selective it's very easy to go go from 100 to a th000 a th000 to go go from 100 to a th000 a th000 to go go from 100 to a th000 a th000 to 10,000 without paying attention to 10,000 without paying attention to 10,000 without paying attention to making sure everyone has a unified making sure everyone has a unified making sure everyone has a unified purpose it's so powerful if your company purpose it's so powerful if your company purpose it's so powerful if your company consists of a lot of different feif that consists of a lot of different feif that consists of a lot of different feif that all want to do their own thing they're all want to do their own thing they're all want to do their own thing they're all optimizing for their own thing um uh all optimizing for their own thing um uh all optimizing for their own thing um uh it's very hard to get anything done but it's very hard to get anything done but it's very hard to get anything done but if everyone sees the broader purpose of if everyone sees the broader purpose of if everyone sees the broader purpose of the company if there's trust and there's the company if there's trust and there's the company if there's trust and there's dedication to doing the right thing that dedication to doing the right thing that dedication to doing the right thing that is a superpower that in itself I think is a superpower that in itself I think is a superpower that in itself I think can overcome almost every other can overcome almost every other can overcome almost every other disadvantage and you know it's to Steve disadvantage and you know it's to Steve disadvantage and you know it's to Steve Jobs a players a players want to look Jobs a players a players want to look Jobs a players a players want to look around and see other a players is around and see other a players is around and see other a players is another way of of saying I don't know another way of of saying I don't know another way of of saying I don't know what that is about human nature but it what that is about human nature but it what that is about human nature but it is demotivating to see people who are is demotivating to see people who are is demotivating to see people who are not obsessively driving towards a not obsessively driving towards a not obsessively driving towards a singular Mission and it is on the flip singular Mission and it is on the flip singular Mission and it is on the flip side of that super motivating to see side of that super motivating to see side of that super motivating to see that it's interesting uh what's it take that it's interesting uh what's it take that it's interesting uh what's it take to be a great AI researcher or engineer

to be a great AI researcher or engineer to be a great AI researcher or engineer from everything you've seen from working from everything you've seen from working from everything you've seen from working with so many amazing people yeah um I with so many amazing people yeah um I with so many amazing people yeah um I think the number one quality especially think the number one quality especially think the number one quality especially on the research side but really both is on the research side but really both is on the research side but really both is open-mindedness sounds easy to be open-mindedness sounds easy to be open-mindedness sounds easy to be open-minded right you're just like oh open-minded right you're just like oh open-minded right you're just like oh I'm open to anything um but you know if I'm open to anything um but you know if I'm open to anything um but you know if I if I think about my own early history I if I think about my own early history I if I think about my own early history in the scaling hypothesis um I was in the scaling hypothesis um I was in the scaling hypothesis um I was seeing the same data others were seeing seeing the same data others were seeing seeing the same data others were seeing I don't think I was like a better I don't think I was like a better I don't think I was like a better programmer or better at coming up with programmer or better at coming up with programmer or better at coming up with research ideas than any of the hundreds research ideas than any of the hundreds research ideas than any of the hundreds of people that I worked with um in some of people that I worked with um in some of people that I worked with um in some ways in some ways I was worse um uh you ways in some ways I was worse um uh you ways in some ways I was worse um uh you know like i' I've never like you know know like i' I've never like you know know like i' I've never like you know precise programming of like you know precise programming of like you know precise programming of like you know finding the bug writing the GPU kernels finding the bug writing the GPU kernels finding the bug writing the GPU kernels like I could point you to a 100 people like I could point you to a 100 people like I could point you to a 100 people here who are better who are better at here who are better who are better at here who are better who are better at that than I am um but but the the thing that than I am um but but the the thing that than I am um but but the the thing that that that I think I did have that that that that I think I did have that that that that I think I did have that was different was that I was just was different was that I was just was different was that I was just willing to look at something with new willing to look at something with new willing to look at something with new eyes right people said oh you know we eyes right people said oh you know we eyes right people said oh you know we don't have the right algorithms yet we don't have the right algorithms yet we don't have the right algorithms yet we haven't come up with the right the right haven't come up with the right the right haven't come up with the right the right way to do things and I was just like uh way to do things and I was just like uh way to do things and I was just like uh I don't know like you know this neural I don't know like you know this neural I don't know like you know this neural net has like 30 billion 30 million net has like 30 billion 30 million net has like 30 billion 30 million parameters like what if we gave it 50 parameters like what if we gave it 50 parameters like what if we gave it 50 million instead like let's plot some million instead like let's plot some million instead like let's plot some graphs like that that basic scientific graphs like that that basic scientific graphs like that that basic scientific mindset of like oh man like I I I just I mindset of like oh man like I I I just I mindset of like oh man like I I I just I just like I you know I see some variable just like I you know I see some variable just like I you know I see some variable that I could change like what happens that I could change like what happens that I could change like what happens when it changes like let's let's try when it changes like let's let's try when it changes like let's let's try these different things and like create a these different things and like create a these different things and like create a graph for even this this was like the

graph for even this this was like the graph for even this this was like the simplest thing in the world right change simplest thing in the world right change simplest thing in the world right change the number of you know this wasn't like the number of you know this wasn't like the number of you know this wasn't like PhD level experimental design this was PhD level experimental design this was PhD level experimental design this was like this was like simple and stupid like this was like simple and stupid like this was like simple and stupid like anyone could have done this if you like anyone could have done this if you like anyone could have done this if you if you just told them that that that it if you just told them that that that it if you just told them that that that it was important it's also not hard to was important it's also not hard to was important it's also not hard to understand you didn't need to be understand you didn't need to be understand you didn't need to be brilliant to come up with this um but brilliant to come up with this um but brilliant to come up with this um but you put the two things together and you you put the two things together and you you put the two things together and you know some tiny number of people some know some tiny number of people some know some tiny number of people some singled digigit number of people have singled digigit number of people have singled digigit number of people have have driven forward the whole field by have driven forward the whole field by have driven forward the whole field by realizing this uh and and it's you know realizing this uh and and it's you know realizing this uh and and it's you know it's often like that if you look back at it's often like that if you look back at it's often like that if you look back at the Discover you know the discoveries in the Discover you know the discoveries in the Discover you know the discoveries in in in history they're they're often like in in history they're they're often like in in history they're they're often like that and so this this open-mindedness that and so this this open-mindedness that and so this this open-mindedness and this willingness to see with new and this willingness to see with new and this willingness to see with new eyes that often comes from being newer eyes that often comes from being newer eyes that often comes from being newer to the field often experience is a to the field often experience is a to the field often experience is a disadvantage for this that is the most disadvantage for this that is the most disadvantage for this that is the most important thing it's very hard to look important thing it's very hard to look important thing it's very hard to look for and test for but I think I think for and test for but I think I think for and test for but I think I think it's the most important thing because it's the most important thing because it's the most important thing because when you when you find something some when you when you find something some when you when you find something some really new way of thinking thinking really new way of thinking thinking really new way of thinking thinking about things when you have the about things when you have the about things when you have the initiative to do that it's absolutely initiative to do that it's absolutely initiative to do that it's absolutely transformative and also be able to do transformative and also be able to do transformative and also be able to do kind of Rapid experimentation and in the kind of Rapid experimentation and in the kind of Rapid experimentation and in the face of that be open-minded and curious face of that be open-minded and curious face of that be open-minded and curious and looking at the data from just these and looking at the data from just these and looking at the data from just these fresh eyes and see what is that actually fresh eyes and see what is that actually fresh eyes and see what is that actually saying that applies in uh mechanistic saying that applies in uh mechanistic saying that applies in uh mechanistic interpretability it's another example of interpretability it's another example of interpretability it's another example of this like some of the early work in this like some of the early work in this like some of the early work in mechanistic interpretability so simple mechanistic interpretability so simple mechanistic interpretability so simple it's it's just no one thought to care it's it's just no one thought to care it's it's just no one thought to care about this question before you said what about this question before you said what about this question before you said what it takes to be a great AI researcher can it takes to be a great AI researcher can it takes to be a great AI researcher can we rewind the clock back what what we rewind the clock back what what we rewind the clock back what what advice would you give to people advice would you give to people advice would you give to people interested in AI they're young looking interested in AI they're young looking interested in AI they're young looking forward how can I make an impact on the forward how can I make an impact on the forward how can I make an impact on the world I think my number one piece of

world I think my number one piece of world I think my number one piece of advice is to just start playing with the advice is to just start playing with the advice is to just start playing with the models um this was actually I I I worry models um this was actually I I I worry models um this was actually I I I worry a little this seems like obvious advice a little this seems like obvious advice a little this seems like obvious advice now I think three years ago it wasn't now I think three years ago it wasn't now I think three years ago it wasn't obvious and people started by oh let me obvious and people started by oh let me obvious and people started by oh let me read the latest reinforcement learning read the latest reinforcement learning read the latest reinforcement learning paper let me you know let me let me kind paper let me you know let me let me kind paper let me you know let me let me kind of um no I mean that was really the that of um no I mean that was really the that of um no I mean that was really the that was really the the and I mean you should was really the the and I mean you should was really the the and I mean you should do that as well but uh now you know with do that as well but uh now you know with do that as well but uh now you know with wider availability of models and apis wider availability of models and apis wider availability of models and apis people are doing this more but I think I people are doing this more but I think I people are doing this more but I think I think just experiential knowledge um think just experiential knowledge um think just experiential knowledge um these models are new artifacts that no these models are new artifacts that no these models are new artifacts that no one really understands um and so getting one really understands um and so getting one really understands um and so getting experience playing with them I would experience playing with them I would experience playing with them I would also say again in line with the like do also say again in line with the like do also say again in line with the like do something new think in some new something new think in some new something new think in some new Direction like there are all these Direction like there are all these Direction like there are all these things that haven't been explored like things that haven't been explored like things that haven't been explored like for example mechanistic interpretability for example mechanistic interpretability for example mechanistic interpretability is still very new it's probably better is still very new it's probably better is still very new it's probably better to work on that than it is to work on to work on that than it is to work on to work on that than it is to work on new model architectures because it's you new model architectures because it's you new model architectures because it's you know it's more popular than it was know it's more popular than it was know it's more popular than it was before there are probably like a hundred before there are probably like a hundred before there are probably like a hundred people working on it but there aren't people working on it but there aren't people working on it but there aren't like 10,000 people working on it and like 10,000 people working on it and like 10,000 people working on it and it's it's just this just this this it's it's just this just this this it's it's just this just this this fertile area for study like like you fertile area for study like like you fertile area for study like like you know it's there's there's so much like know it's there's there's so much like know it's there's there's so much like low hangen fruit you can just walk by low hangen fruit you can just walk by low hangen fruit you can just walk by and you know you can just walk by and and you know you can just walk by and and you know you can just walk by and you can pick things um and and the the you can pick things um and and the the you can pick things um and and the the the only reason for whatever reason the only reason for whatever reason the only reason for whatever reason people aren't people aren't interested people aren't people aren't interested people aren't people aren't interested in it enough I think there are some in it enough I think there are some in it enough I think there are some things around long long Horizon learning things around long long Horizon learning things around long long Horizon learning and long Horizon tasks where there's a and long Horizon tasks where there's a and long Horizon tasks where there's a lot to be done I think evaluations are lot to be done I think evaluations are lot to be done I think evaluations are still we're still very early in our still we're still very early in our still we're still very early in our ability to study evaluations

ability to study evaluations ability to study evaluations particularly for dynamic systems acting particularly for dynamic systems acting particularly for dynamic systems acting in the world I think there's some stuff in the world I think there's some stuff in the world I think there's some stuff around around around multi-agent um skate where the puck is multi-agent um skate where the puck is multi-agent um skate where the puck is going is my is my advice and you don't going is my is my advice and you don't going is my is my advice and you don't have to be brilliant to think of it like have to be brilliant to think of it like have to be brilliant to think of it like all the things that are going to be all the things that are going to be all the things that are going to be exciting in 5 years like in in people exciting in 5 years like in in people exciting in 5 years like in in people even mention them as like you know even mention them as like you know even mention them as like you know conventional wisdom but like it's it's conventional wisdom but like it's it's conventional wisdom but like it's it's just somehow there's this barrier that just somehow there's this barrier that just somehow there's this barrier that people don't people don't double down as people don't people don't double down as people don't people don't double down as much as they could or they're afraid to much as they could or they're afraid to much as they could or they're afraid to do something that's not the popular do something that's not the popular do something that's not the popular thing I don't know why it happens but thing I don't know why it happens but thing I don't know why it happens but like getting over that barrier is the like getting over that barrier is the like getting over that barrier is the that's the my number one piece of advice that's the my number one piece of advice that's the my number one piece of advice let's talk if we could a bit about let's talk if we could a bit about let's talk if we could a bit about posttraining yeah so it uh seems that posttraining yeah so it uh seems that posttraining yeah so it uh seems that the modern posttraining the modern posttraining the modern posttraining recipe has uh a little bit of everything recipe has uh a little bit of everything recipe has uh a little bit of everything so supervised fine tuning so supervised fine tuning so supervised fine tuning rhf uh the the the Constitutional AI rhf uh the the the Constitutional AI rhf uh the the the Constitutional AI with RL a if best acronym it's again with RL a if best acronym it's again with RL a if best acronym it's again that naming that naming that naming thing uh and then synthetic data seems thing uh and then synthetic data seems thing uh and then synthetic data seems like a lot of synthetic data or at least like a lot of synthetic data or at least like a lot of synthetic data or at least trying to figure out ways to have high trying to figure out ways to have high trying to figure out ways to have high quality synthetic data so what's the uh quality synthetic data so what's the uh quality synthetic data so what's the uh if this is a secret sauce that makes if this is a secret sauce that makes if this is a secret sauce that makes anthropic claw so uh incredible what how anthropic claw so uh incredible what how anthropic claw so uh incredible what how how much of the magic is in the how much of the magic is in the how much of the magic is in the pre-training how much much of is in the pre-training how much much of is in the pre-training how much much of is in the post training yeah um I mean so first of post training yeah um I mean so first of post training yeah um I mean so first of all we're not perfectly able to measure all we're not perfectly able to measure all we're not perfectly able to measure that ourselves um uh you know when you that ourselves um uh you know when you that ourselves um uh you know when you see some some great character ability see some some great character ability see some some great character ability sometimes it's hard to tell whether it sometimes it's hard to tell whether it sometimes it's hard to tell whether it came from pre-training or post-training came from pre-training or post-training came from pre-training or post-training uh we developed ways to try and uh we developed ways to try and uh we developed ways to try and distinguish between those two but distinguish between those two but distinguish between those two but they're not perfect you know the second

they're not perfect you know the second they're not perfect you know the second thing I would say is you know it's when thing I would say is you know it's when thing I would say is you know it's when there is an advantage and I think we've there is an advantage and I think we've there is an advantage and I think we've been pretty good at in general in been pretty good at in general in been pretty good at in general in general at RL Perhaps Perhaps the best general at RL Perhaps Perhaps the best general at RL Perhaps Perhaps the best although although I don't know because I although although I don't know because I although although I don't know because I don't see what goes on inside other don't see what goes on inside other don't see what goes on inside other companies uh companies uh companies uh usually it isn't oh my God we have this usually it isn't oh my God we have this usually it isn't oh my God we have this secret magic method that others don't secret magic method that others don't secret magic method that others don't have right usually it's like well you have right usually it's like well you have right usually it's like well you know we got better at the infrastructure know we got better at the infrastructure know we got better at the infrastructure so we could run it for longer or you so we could run it for longer or you so we could run it for longer or you know we were able to get higher quality know we were able to get higher quality know we were able to get higher quality data or we were able to filter our data data or we were able to filter our data data or we were able to filter our data better or we able to you know combine better or we able to you know combine better or we able to you know combine these methods and practice it's it's these methods and practice it's it's these methods and practice it's it's usually some boring matter of matter of usually some boring matter of matter of usually some boring matter of matter of kind of uh practice and tradecraft um so kind of uh practice and tradecraft um so kind of uh practice and tradecraft um so you know when I think about how to do you know when I think about how to do you know when I think about how to do something special in terms of how we something special in terms of how we something special in terms of how we train these models both pre-training but train these models both pre-training but train these models both pre-training but even more so posttraining um you know I even more so posttraining um you know I even more so posttraining um you know I I I really think of it a little more I I really think of it a little more I I really think of it a little more again as like designing airplanes or again as like designing airplanes or again as like designing airplanes or cars like you know it's not just like oh cars like you know it's not just like oh cars like you know it's not just like oh man I have the BL blueprint like maybe man I have the BL blueprint like maybe man I have the BL blueprint like maybe that makes you make the next airplane that makes you make the next airplane that makes you make the next airplane but like there's some there's some but like there's some there's some but like there's some there's some cultural tradecraft of how we think cultural tradecraft of how we think cultural tradecraft of how we think about the design process that I think is about the design process that I think is about the design process that I think is more important than than you know than more important than than you know than more important than than you know than than any particular Gizmo were able to than any particular Gizmo were able to than any particular Gizmo were able to invent okay well about let me ask you invent okay well about let me ask you invent okay well about let me ask you about specific techniques so first on about specific techniques so first on about specific techniques so first on rhf what do you think think just zooming rhf what do you think think just zooming rhf what do you think think just zooming out intuition almost philosophy why do out intuition almost philosophy why do out intuition almost philosophy why do you think rhf works so well if I go back you think rhf works so well if I go back you think rhf works so well if I go back to like the scaling hypothesis one of to like the scaling hypothesis one of to like the scaling hypothesis one of the ways to skate the scaling hypothesis the ways to skate the scaling hypothesis the ways to skate the scaling hypothesis is if you train for x and you throw is if you train for x and you throw is if you train for x and you throw enough compute at it um then you get X enough compute at it um then you get X enough compute at it um then you get X and and so rlf is good at doing what

and and so rlf is good at doing what and and so rlf is good at doing what humans want the model to do or at least humans want the model to do or at least humans want the model to do or at least um to State it more precisely doing what um to State it more precisely doing what um to State it more precisely doing what humans who look at the model for a brief humans who look at the model for a brief humans who look at the model for a brief period of time and consider different period of time and consider different period of time and consider different possible responses what prefer as the possible responses what prefer as the possible responses what prefer as the response uh which is not perfect from response uh which is not perfect from response uh which is not perfect from both a safety and capabilities both a safety and capabilities both a safety and capabilities perspective in that humans are are often perspective in that humans are are often perspective in that humans are are often not able to perfectly identify what the not able to perfectly identify what the not able to perfectly identify what the model wants and what humans want in the model wants and what humans want in the model wants and what humans want in the moment may not be what they want in the moment may not be what they want in the moment may not be what they want in the long term so there's there's a lot of long term so there's there's a lot of long term so there's there's a lot of subtlety there but the models are good subtlety there but the models are good subtlety there but the models are good at uh you know producing what the humans at uh you know producing what the humans at uh you know producing what the humans in some shallow sense want uh and it in some shallow sense want uh and it in some shallow sense want uh and it actually turns out that you don't even actually turns out that you don't even actually turns out that you don't even have to throw that much compute at it have to throw that much compute at it have to throw that much compute at it because of another thing which is this because of another thing which is this because of another thing which is this this thing about a strong pre-trained this thing about a strong pre-trained this thing about a strong pre-trained model being halfway to anywhere uh uh uh model being halfway to anywhere uh uh uh model being halfway to anywhere uh uh uh so once you have the pre-trained model so once you have the pre-trained model so once you have the pre-trained model you have all the representations you you have all the representations you you have all the representations you need to to get the model uh to get the need to to get the model uh to get the need to to get the model uh to get the model where you where you want it to go model where you where you want it to go model where you where you want it to go so do you think so do you think so do you think rhf makes the model smarter or just rhf makes the model smarter or just rhf makes the model smarter or just appears smarter to the humans I don't appears smarter to the humans I don't appears smarter to the humans I don't think it makes the model smarter I don't think it makes the model smarter I don't think it makes the model smarter I don't think it just makes the model appear think it just makes the model appear think it just makes the model appear smarter it's like smarter it's like smarter it's like rhf like Bridges the gap between the rhf like Bridges the gap between the rhf like Bridges the gap between the human and the model right I could have human and the model right I could have human and the model right I could have something really smart that like can't something really smart that like can't something really smart that like can't communicate at all right we all know communicate at all right we all know communicate at all right we all know people like this um people who are people like this um people who are people like this um people who are really smart but that you know can't really smart but that you know can't really smart but that you know can't understand what they're saying um uh so understand what they're saying um uh so understand what they're saying um uh so I think I think rhf just bridges that I think I think rhf just bridges that I think I think rhf just bridges that Gap um I I think it's not it's not the Gap um I I think it's not it's not the Gap um I I think it's not it's not the only kind of RL we do it's not the only only kind of RL we do it's not the only only kind of RL we do it's not the only kind of RL that will happen in the kind of RL that will happen in the kind of RL that will happen in the future I think RL has the potential to

future I think RL has the potential to future I think RL has the potential to make models smarter to make them reason make models smarter to make them reason make models smarter to make them reason better to make them operate better to better to make them operate better to better to make them operate better to make them develop new skills even and make them develop new skills even and make them develop new skills even and perhaps that could be done you know even perhaps that could be done you know even perhaps that could be done you know even in some cases with human feedback but in some cases with human feedback but in some cases with human feedback but the kind of rhf we we do today mostly the kind of rhf we we do today mostly the kind of rhf we we do today mostly doesn't do that yet although we're very doesn't do that yet although we're very doesn't do that yet although we're very quickly starting to be able to but it it quickly starting to be able to but it it quickly starting to be able to but it it appears to sort of increase if you look appears to sort of increase if you look appears to sort of increase if you look at the metric of helpfulness it at the metric of helpfulness it at the metric of helpfulness it increases that it also increases what increases that it also increases what increases that it also increases what was this this word in Leopold's essay un was this this word in Leopold's essay un was this this word in Leopold's essay un hobbling where basically the models are hobbling where basically the models are hobbling where basically the models are hobbled and then you do various hobbled and then you do various hobbled and then you do various trainings to them to un hobble them so I trainings to them to un hobble them so I trainings to them to un hobble them so I I know I like that word because it's I know I like that word because it's I know I like that word because it's like a rare word but so so I think rhf like a rare word but so so I think rhf like a rare word but so so I think rhf un hobbles the models in some ways un hobbles the models in some ways un hobbles the models in some ways um and then there are other ways where M um and then there are other ways where M um and then there are other ways where M hasn't yet been un hobbled and and you hasn't yet been un hobbled and and you hasn't yet been un hobbled and and you know needs to needs to un hobble if you know needs to needs to un hobble if you know needs to needs to un hobble if you can say in terms of cost is pre-training can say in terms of cost is pre-training can say in terms of cost is pre-training the most expensive thing or is the most expensive thing or is the most expensive thing or is post-training creep up to that at the post-training creep up to that at the post-training creep up to that at the present moment it is still the case that present moment it is still the case that present moment it is still the case that uh pre-training is the majority of the uh pre-training is the majority of the uh pre-training is the majority of the cost I don't know what to expect in the cost I don't know what to expect in the cost I don't know what to expect in the future but I could certainly anticipate future but I could certainly anticipate future but I could certainly anticipate a future where post-training is the a future where post-training is the a future where post-training is the majority of the cost in that future you majority of the cost in that future you majority of the cost in that future you anticipate would it be the humans or the anticipate would it be the humans or the anticipate would it be the humans or the AI That's the costly thing for the Post AI That's the costly thing for the Post AI That's the costly thing for the Post training I I I I I don't think you can training I I I I I don't think you can training I I I I I don't think you can scale up humans enough to get high scale up humans enough to get high scale up humans enough to get high quality any any kind of method that quality any any kind of method that quality any any kind of method that relies on humans and uses a large amount relies on humans and uses a large amount relies on humans and uses a large amount of compute it's going to have to rely on of compute it's going to have to rely on of compute it's going to have to rely on some scaled supervision method like uh some scaled supervision method like uh some scaled supervision method like uh uh like um it you know debate or uh like um it you know debate or uh like um it you know debate or iterated amplification or something like iterated amplification or something like iterated amplification or something like that so on that so on that so on that super interesting um set of ideas

that super interesting um set of ideas that super interesting um set of ideas around constitutional AI can describe around constitutional AI can describe around constitutional AI can describe what it is as first detailed in December what it is as first detailed in December what it is as first detailed in December 2022 paper and uh and be on that what is 2022 paper and uh and be on that what is 2022 paper and uh and be on that what is it yes so this was from two years ago it yes so this was from two years ago it yes so this was from two years ago the basic idea is so we describe what the basic idea is so we describe what the basic idea is so we describe what rhf is you have uh you have a model and rhf is you have uh you have a model and rhf is you have uh you have a model and uh it you know spits out two you know uh it you know spits out two you know uh it you know spits out two you know like you just sample from it twice it like you just sample from it twice it like you just sample from it twice it spits out two possible responses and spits out two possible responses and spits out two possible responses and you're like human which response you you're like human which response you you're like human which response you like better or another variant of it is like better or another variant of it is like better or another variant of it is rate this response on a scale of 1 to rate this response on a scale of 1 to rate this response on a scale of 1 to seven so that's hard because you need to seven so that's hard because you need to seven so that's hard because you need to scale up human interaction and uh it's scale up human interaction and uh it's scale up human interaction and uh it's very implicit right I don't have a sense very implicit right I don't have a sense very implicit right I don't have a sense of what I what I want the model to do I of what I what I want the model to do I of what I what I want the model to do I just have a sense of like what this just have a sense of like what this just have a sense of like what this average of a thousand humans wants the average of a thousand humans wants the average of a thousand humans wants the model to do so two ideas one is could model to do so two ideas one is could model to do so two ideas one is could the AI system itself decide which uh the AI system itself decide which uh the AI system itself decide which uh which response is better right could you which response is better right could you which response is better right could you show the AI system these two responses show the AI system these two responses show the AI system these two responses and and ask which which which response and and ask which which which response and and ask which which which response is better and then second well what is better and then second well what is better and then second well what Criterion should the AI use and so then Criterion should the AI use and so then Criterion should the AI use and so then there's this idea because you have a there's this idea because you have a there's this idea because you have a single document a constitution if you single document a constitution if you single document a constitution if you will that says these are the principles will that says these are the principles will that says these are the principles the model should be using to to respond the model should be using to to respond the model should be using to to respond and the AI system reads those um it and the AI system reads those um it and the AI system reads those um it reads those principles as well as reads those principles as well as reads those principles as well as reading the environment and the response reading the environment and the response reading the environment and the response and it says well how good did the AI and it says well how good did the AI and it says well how good did the AI model do um it's basically a form of model do um it's basically a form of model do um it's basically a form of self-play you you're kind of training self-play you you're kind of training self-play you you're kind of training the model against itself and so the AI the model against itself and so the AI the model against itself and so the AI gives the response and then you feed gives the response and then you feed gives the response and then you feed that back into What's called the

that back into What's called the that back into What's called the preference model which in turn feeds the preference model which in turn feeds the preference model which in turn feeds the model to make it better um so you have model to make it better um so you have model to make it better um so you have this triangle of like the AI the this triangle of like the AI the this triangle of like the AI the preference model and the Improvement of preference model and the Improvement of preference model and the Improvement of the AI itself and we should say that in the AI itself and we should say that in the AI itself and we should say that in the Constitution the set of principles the Constitution the set of principles the Constitution the set of principles are like human interpretable they're are like human interpretable they're are like human interpretable they're like yeah yeah it's something both the like yeah yeah it's something both the like yeah yeah it's something both the human and the AI system can read so it human and the AI system can read so it human and the AI system can read so it has this nice this nice kind of has this nice this nice kind of has this nice this nice kind of translatability or symmetry um you know translatability or symmetry um you know translatability or symmetry um you know in in practice we both use a model in in practice we both use a model in in practice we both use a model Constitution and we use rhf and we use Constitution and we use rhf and we use Constitution and we use rhf and we use some of these other methods so it's it's some of these other methods so it's it's some of these other methods so it's it's turned into one tool in a in a toolkit turned into one tool in a in a toolkit turned into one tool in a in a toolkit that both reduces the need for rhf and that both reduces the need for rhf and that both reduces the need for rhf and increases the value we get from um from increases the value we get from um from increases the value we get from um from from using each data point of R lhf um from using each data point of R lhf um from using each data point of R lhf um it also interacts in interesting ways it also interacts in interesting ways it also interacts in interesting ways with kind of future reasoning type RL with kind of future reasoning type RL with kind of future reasoning type RL methods so um it's it's one tool in the methods so um it's it's one tool in the methods so um it's it's one tool in the toolkit but but I I think it is a very toolkit but but I I think it is a very toolkit but but I I think it is a very important tool well it's a compelling important tool well it's a compelling important tool well it's a compelling one to us humans you know thinking about one to us humans you know thinking about one to us humans you know thinking about the founding fathers and the founding of the founding fathers and the founding of the founding fathers and the founding of the United the United the United States the natural question is who and States the natural question is who and States the natural question is who and how do you think it gets to define the how do you think it gets to define the how do you think it gets to define the constitution the the set of principles constitution the the set of principles constitution the the set of principles in the Constitution yeah so I'll give in the Constitution yeah so I'll give in the Constitution yeah so I'll give like a practical um answer and a more like a practical um answer and a more like a practical um answer and a more abstract answer I think the Practical abstract answer I think the Practical abstract answer I think the Practical answer is like look in practice models answer is like look in practice models answer is like look in practice models get used by all kinds of different like get used by all kinds of different like get used by all kinds of different like customers right and and so uh you can customers right and and so uh you can customers right and and so uh you can have this idea where you know the model have this idea where you know the model have this idea where you know the model can can have specialized rules or can can have specialized rules or can can have specialized rules or principles you know we fine-tune principles you know we fine-tune principles you know we fine-tune versions of models implicitly we've versions of models implicitly we've versions of models implicitly we've talked about doing it explicitly having talked about doing it explicitly having talked about doing it explicitly having having special principles that people

having special principles that people having special principles that people can can build into the models um uh so can can build into the models um uh so can can build into the models um uh so from a practical perspective the answer from a practical perspective the answer from a practical perspective the answer can be very different from different can be very different from different can be very different from different people uh you know customers service people uh you know customers service people uh you know customers service agent uh you know behaves very agent uh you know behaves very agent uh you know behaves very differently from a lawyer and obeys differently from a lawyer and obeys differently from a lawyer and obeys different principles um but I think at different principles um but I think at different principles um but I think at the base of it there are specific the base of it there are specific the base of it there are specific principles that the models uh you know principles that the models uh you know principles that the models uh you know have to obey I think a lot of them are have to obey I think a lot of them are have to obey I think a lot of them are things that people would agree with things that people would agree with things that people would agree with everyone agrees that you know we don't everyone agrees that you know we don't everyone agrees that you know we don't you know we don't want models to present you know we don't want models to present you know we don't want models to present these cbrn risks um I think we can go a these cbrn risks um I think we can go a these cbrn risks um I think we can go a little further and agree with some basic little further and agree with some basic little further and agree with some basic principles of democracy and the rule of principles of democracy and the rule of principles of democracy and the rule of law beyond that it gets you know very law beyond that it gets you know very law beyond that it gets you know very uncertain and and there our goal is uncertain and and there our goal is uncertain and and there our goal is generally for the models to be more generally for the models to be more generally for the models to be more neutral to not espouse a particular neutral to not espouse a particular neutral to not espouse a particular point of view and you know more just be point of view and you know more just be point of view and you know more just be kind of like wise uh agents or advisers kind of like wise uh agents or advisers kind of like wise uh agents or advisers that will help you think things through that will help you think things through that will help you think things through and will you know present present and will you know present present and will you know present present possible considerations but you know possible considerations but you know possible considerations but you know don't express you know stronger specific don't express you know stronger specific don't express you know stronger specific opinions open AI released a model spec opinions open AI released a model spec opinions open AI released a model spec where it kind of clearly concretely where it kind of clearly concretely where it kind of clearly concretely defines some of the goals of the model defines some of the goals of the model defines some of the goals of the model and specific examples like AB how the and specific examples like AB how the and specific examples like AB how the model should behave do you find that model should behave do you find that model should behave do you find that interesting by the way I should mention interesting by the way I should mention interesting by the way I should mention the I believe the brilliant John the I believe the brilliant John the I believe the brilliant John Schulman was a part of that he's now an Schulman was a part of that he's now an Schulman was a part of that he's now an anthropic uh do you think this is a anthropic uh do you think this is a anthropic uh do you think this is a useful Direction might anthropic release useful Direction might anthropic release useful Direction might anthropic release a model spec as well yeah so I think a model spec as well yeah so I think a model spec as well yeah so I think that's a pretty useful direction again that's a pretty useful direction again that's a pretty useful direction again it has a lot in common with uh it has a lot in common with uh it has a lot in common with uh constitutional AI so again another constitutional AI so again another constitutional AI so again another example of like a race to the top right

example of like a race to the top right example of like a race to the top right we have something that's like we think we have something that's like we think we have something that's like we think you know a better and more responsible you know a better and more responsible you know a better and more responsible way of doing things um it's also a way of doing things um it's also a way of doing things um it's also a competitive advantage um then uh others competitive advantage um then uh others competitive advantage um then uh others kind of you know discover that it has kind of you know discover that it has kind of you know discover that it has advantages and then start to do that advantages and then start to do that advantages and then start to do that thing uh we then no longer have the thing uh we then no longer have the thing uh we then no longer have the competitive Advantage but it's good from competitive Advantage but it's good from competitive Advantage but it's good from the perspective that now everyone has the perspective that now everyone has the perspective that now everyone has adopted a positive practice that others adopted a positive practice that others adopted a positive practice that others were not adopting and so our response to were not adopting and so our response to were not adopting and so our response to that as well looks like we need a new that as well looks like we need a new that as well looks like we need a new competitive advantage in order to keep competitive advantage in order to keep competitive advantage in order to keep driving this race upwards um so that's driving this race upwards um so that's driving this race upwards um so that's that's how I generally feel about that I that's how I generally feel about that I that's how I generally feel about that I also think every implementation of these also think every implementation of these also think every implementation of these things is different so you know there things is different so you know there things is different so you know there were some things in the model spec that were some things in the model spec that were some things in the model spec that were not in constitutional Ai and so you were not in constitutional Ai and so you were not in constitutional Ai and so you know we you know we can always we can know we you know we can always we can know we you know we can always we can always adopt those things or you know at always adopt those things or you know at always adopt those things or you know at least learn from them um so again I least learn from them um so again I least learn from them um so again I think this is an example of like the think this is an example of like the think this is an example of like the positive Dynamic that uh that that that positive Dynamic that uh that that that positive Dynamic that uh that that that I that that I think we should all want I that that I think we should all want I that that I think we should all want the field to have let's talk about the the field to have let's talk about the the field to have let's talk about the incredible ESS Machines of love and incredible ESS Machines of love and incredible ESS Machines of love and grace I recommend everybody read it it's grace I recommend everybody read it it's grace I recommend everybody read it it's a long one it is rather long yeah it's a long one it is rather long yeah it's a long one it is rather long yeah it's really refreshing to read concrete ideas really refreshing to read concrete ideas really refreshing to read concrete ideas about what a positive future looks like about what a positive future looks like about what a positive future looks like and you took sort of a bold stance and you took sort of a bold stance and you took sort of a bold stance because like it's very possible you because like it's very possible you because like it's very possible you might be wrong on the dates or specific might be wrong on the dates or specific might be wrong on the dates or specific applications yeah I'm fully expecting to applications yeah I'm fully expecting to applications yeah I'm fully expecting to you know to definitely be wrong about you know to definitely be wrong about you know to definitely be wrong about all the details I might be be just all the details I might be be just all the details I might be be just spectacularly wrong about the whole spectacularly wrong about the whole spectacularly wrong about the whole thing and people will you know will thing and people will you know will thing and people will you know will laugh at me for years um uh that's laugh at me for years um uh that's laugh at me for years um uh that's that's how that's that's just how the that's how that's that's just how the that's how that's that's just how the future works so you provided a bunch of future works so you provided a bunch of future works so you provided a bunch of concrete positive impacts of AI and how concrete positive impacts of AI and how concrete positive impacts of AI and how you know exactly a super intelligent AI

you know exactly a super intelligent AI you know exactly a super intelligent AI might accelerate the rate of might accelerate the rate of might accelerate the rate of breakthroughs in for example biology and breakthroughs in for example biology and breakthroughs in for example biology and chemistry that would then lead to things chemistry that would then lead to things chemistry that would then lead to things like we cure most cancers prevent all like we cure most cancers prevent all like we cure most cancers prevent all infectious disease double the human infectious disease double the human infectious disease double the human lifespan and so on so let's talk about lifespan and so on so let's talk about lifespan and so on so let's talk about this essay first can you give a high this essay first can you give a high this essay first can you give a high level vision of this essay and um what level vision of this essay and um what level vision of this essay and um what key takeaways that people should have key takeaways that people should have key takeaways that people should have yeah I have spent a lot of time and yeah I have spent a lot of time and yeah I have spent a lot of time and anthropic has spent a lot of effort on anthropic has spent a lot of effort on anthropic has spent a lot of effort on like you know how do we address the like you know how do we address the like you know how do we address the risks of AI right how do we think about risks of AI right how do we think about risks of AI right how do we think about those risks like we're trying to do a those risks like we're trying to do a those risks like we're trying to do a race to the top you know that requires race to the top you know that requires race to the top you know that requires us to build all these capabilities and us to build all these capabilities and us to build all these capabilities and the abilities are cool but you know you the abilities are cool but you know you the abilities are cool but you know you know we're we're we're like a big part know we're we're we're like a big part know we're we're we're like a big part of what we're trying to do is like is of what we're trying to do is like is of what we're trying to do is like is like address the risks and the like address the risks and the like address the risks and the justification for that is like well you justification for that is like well you justification for that is like well you know all these positive things you know know all these positive things you know know all these positive things you know the the market is this very healthy the the market is this very healthy the the market is this very healthy organism right it's going to produce all organism right it's going to produce all organism right it's going to produce all the positive things the risks I don't the positive things the risks I don't the positive things the risks I don't know we might mitigate them we might not know we might mitigate them we might not know we might mitigate them we might not and so we can have more impact by trying and so we can have more impact by trying and so we can have more impact by trying to mitigate the risks but I noticed that to mitigate the risks but I noticed that to mitigate the risks but I noticed that one flaw in that way of thinking and one flaw in that way of thinking and one flaw in that way of thinking and it's if not a change in how seriously I it's if not a change in how seriously I it's if not a change in how seriously I take the risks it's it's maybe a change take the risks it's it's maybe a change take the risks it's it's maybe a change in how I talk about them um is that you in how I talk about them um is that you in how I talk about them um is that you know no matter how kind of logical or know no matter how kind of logical or know no matter how kind of logical or rational that line of reasoning that I rational that line of reasoning that I rational that line of reasoning that I just gave might be um if if you kind of just gave might be um if if you kind of just gave might be um if if you kind of only talk about risks your brain only only talk about risks your brain only only talk about risks your brain only thinks about risks and and so I think thinks about risks and and so I think thinks about risks and and so I think it's actually very important to it's actually very important to it's actually very important to understand what if things do go well and

understand what if things do go well and understand what if things do go well and the whole reason we're trying to prevent the whole reason we're trying to prevent the whole reason we're trying to prevent these risks is not because we're afraid these risks is not because we're afraid these risks is not because we're afraid of Technology not because we want to of Technology not because we want to of Technology not because we want to slow it down it's it's it's slow it down it's it's it's slow it down it's it's it's because if we can get to the other side because if we can get to the other side because if we can get to the other side of these risks right if we can run the of these risks right if we can run the of these risks right if we can run the gauntlet successfully um to you know to gauntlet successfully um to you know to gauntlet successfully um to you know to to put it in Stark terms then then on to put it in Stark terms then then on to put it in Stark terms then then on the other side of the gauntlet are all the other side of the gauntlet are all the other side of the gauntlet are all these great things and these things are these great things and these things are these great things and these things are worth fighting for and these things can worth fighting for and these things can worth fighting for and these things can really inspire people and I think I really inspire people and I think I really inspire people and I think I imagine because look you have all these imagine because look you have all these imagine because look you have all these investors all these VCS all these AI investors all these VCS all these AI investors all these VCS all these AI companies talking about all the positive companies talking about all the positive companies talking about all the positive benefits of AI but as you point out it's benefits of AI but as you point out it's benefits of AI but as you point out it's it's it's weird there's actually a dir it's it's weird there's actually a dir it's it's weird there's actually a dir of really getting specific about it of really getting specific about it of really getting specific about it there's a lot of like random people on there's a lot of like random people on there's a lot of like random people on Twitter like posting these kind of like Twitter like posting these kind of like Twitter like posting these kind of like gleaming cities and this this just kind gleaming cities and this this just kind gleaming cities and this this just kind of like Vibe of like grind accelerate of like Vibe of like grind accelerate of like Vibe of like grind accelerate harder like kick out the D you know it's harder like kick out the D you know it's harder like kick out the D you know it's it's just this very this very like it's just this very this very like it's just this very this very like aggressive ideological but then you're aggressive ideological but then you're aggressive ideological but then you're like well what are you what what what like well what are you what what what like well what are you what what what what what are you actually excited about what what are you actually excited about what what are you actually excited about um and so and so I figured that you know um and so and so I figured that you know um and so and so I figured that you know I think it would be interesting and I think it would be interesting and I think it would be interesting and valuable for someone who's actually valuable for someone who's actually valuable for someone who's actually coming from the risk side to to try and coming from the risk side to to try and coming from the risk side to to try and and to try and really make a try at at and to try and really make a try at at and to try and really make a try at at explaining explaining explaining what explaining explaining explaining what explaining explaining explaining what the benefits are um both because I think the benefits are um both because I think the benefits are um both because I think it's something we can all get behind and it's something we can all get behind and it's something we can all get behind and I want people to understand I want them I want people to understand I want them I want people to understand I want them to really understand that this isn't to really understand that this isn't to really understand that this isn't this isn't doomers versus this isn't doomers versus this isn't doomers versus accelerationists um this this

accelerationists um this this accelerationists um this this is that if you have a true understanding is that if you have a true understanding is that if you have a true understanding of of where things are going with with of of where things are going with with of of where things are going with with AI and maybe that's the more important AI and maybe that's the more important AI and maybe that's the more important axis AI is moving fast versus AI is not axis AI is moving fast versus AI is not axis AI is moving fast versus AI is not moving fast then you really appreciate moving fast then you really appreciate moving fast then you really appreciate the benefits and you you you you really the benefits and you you you you really the benefits and you you you you really you want Humanity our civilization to you want Humanity our civilization to you want Humanity our civilization to seize those benefits but you also get seize those benefits but you also get seize those benefits but you also get very serious about anything that could very serious about anything that could very serious about anything that could derail them so I think the starting derail them so I think the starting derail them so I think the starting point is to talk about what this point is to talk about what this point is to talk about what this powerful AI which is the term you like powerful AI which is the term you like powerful AI which is the term you like to use uh most of the world uses AGI but to use uh most of the world uses AGI but to use uh most of the world uses AGI but you don't like the term because it's uh you don't like the term because it's uh you don't like the term because it's uh basically has too much baggage has basically has too much baggage has basically has too much baggage has become meaningless it's like we're stuck become meaningless it's like we're stuck become meaningless it's like we're stuck with the terms like maybe we're stuck with the terms like maybe we're stuck with the terms like maybe we're stuck with the terms and my efforts to change with the terms and my efforts to change with the terms and my efforts to change them are futile it's ADM I'll tell you them are futile it's ADM I'll tell you them are futile it's ADM I'll tell you what else I don't this is like a what else I don't this is like a what else I don't this is like a pointless semantic point but I I I I pointless semantic point but I I I I pointless semantic point but I I I I keep talking about it public so I'm just keep talking about it public so I'm just keep talking about it public so I'm just I'm just going to do it once more um uh I'm just going to do it once more um uh I'm just going to do it once more um uh I I think it's it's a little like like I I think it's it's a little like like I I think it's it's a little like like let's say it was like 1995 and Mor's law let's say it was like 1995 and Mor's law let's say it was like 1995 and Mor's law is making the computers faster and like is making the computers faster and like is making the computers faster and like for some reason there there there there for some reason there there there there for some reason there there there there had been this like verbal tick that like had been this like verbal tick that like had been this like verbal tick that like everyone was like well someday we're everyone was like well someday we're everyone was like well someday we're going to have like super super computers going to have like super super computers going to have like super super computers and like supercomputers are going to be and like supercomputers are going to be and like supercomputers are going to be able to do all these things that like able to do all these things that like able to do all these things that like you know once we have supercomputers you know once we have supercomputers you know once we have supercomputers we'll be able to like sequence the Geno we'll be able to like sequence the Geno we'll be able to like sequence the Geno and we'll be able to do other things and and we'll be able to do other things and and we'll be able to do other things and so and so like one it's true the so and so like one it's true the so and so like one it's true the computers are getting faster and as they computers are getting faster and as they computers are getting faster and as they get faster they're going to be able to get faster they're going to be able to get faster they're going to be able to do all these great things but there's do all these great things but there's do all these great things but there's like there's no discret point at which like there's no discret point at which like there's no discret point at which you had a supercomputer and previous you had a supercomputer and previous you had a supercomputer and previous computers were not to like supercomputer computers were not to like supercomputer computers were not to like supercomputer is a term we use but like it's a vague is a term we use but like it's a vague is a term we use but like it's a vague term to just describe like computers term to just describe like computers term to just describe like computers that are faster than what we have today

that are faster than what we have today that are faster than what we have today um there's no point at which you pass a um there's no point at which you pass a um there's no point at which you pass a threshold and you're like oh my God threshold and you're like oh my God threshold and you're like oh my God we're doing a totally new type of we're doing a totally new type of we're doing a totally new type of computation and new and and so I feel computation and new and and so I feel computation and new and and so I feel that way about AGI like there's just a that way about AGI like there's just a that way about AGI like there's just a smooth exponential and like if if by AGI smooth exponential and like if if by AGI smooth exponential and like if if by AGI you mean like like AI is getting better you mean like like AI is getting better you mean like like AI is getting better and better and like gradually it's going and better and like gradually it's going and better and like gradually it's going to do more and more of what humans do to do more and more of what humans do to do more and more of what humans do until it's going to be smarter than until it's going to be smarter than until it's going to be smarter than humans and then it's going to get humans and then it's going to get humans and then it's going to get smarter even from there then then yes I smarter even from there then then yes I smarter even from there then then yes I believe in AGI if but if if if AGI is believe in AGI if but if if if AGI is believe in AGI if but if if if AGI is some discreet or separate thing which is some discreet or separate thing which is some discreet or separate thing which is the way people often talk about it then the way people often talk about it then the way people often talk about it then it's it's kind of a meaningless buzz it's it's kind of a meaningless buzz it's it's kind of a meaningless buzz word yeah I me to me it's just sort of a word yeah I me to me it's just sort of a word yeah I me to me it's just sort of a IC form of a powerful AI exactly how you IC form of a powerful AI exactly how you IC form of a powerful AI exactly how you define it I mean you define it very define it I mean you define it very define it I mean you define it very nicely so on the intelligence axis it's nicely so on the intelligence axis it's nicely so on the intelligence axis it's just on pure intelligence it's smarter just on pure intelligence it's smarter just on pure intelligence it's smarter than a Nobel Prize winner as you than a Nobel Prize winner as you than a Nobel Prize winner as you describe across most relevant describe across most relevant describe across most relevant disciplines so okay that's just disciplines so okay that's just disciplines so okay that's just intelligence so it's uh both in intelligence so it's uh both in intelligence so it's uh both in creativity and be able to generate new creativity and be able to generate new creativity and be able to generate new ideas all that kind of stuff in every ideas all that kind of stuff in every ideas all that kind of stuff in every discipline Nobel Prize winner okay in discipline Nobel Prize winner okay in discipline Nobel Prize winner okay in their their their prime it can use every modality it so uh prime it can use every modality it so uh prime it can use every modality it so uh that's kind of self-explanatory but just that's kind of self-explanatory but just that's kind of self-explanatory but just operate across all the modalities of the operate across all the modalities of the operate across all the modalities of the world uh it can go off for many hours world uh it can go off for many hours world uh it can go off for many hours days and weeks to do tasks and do its days and weeks to do tasks and do its days and weeks to do tasks and do its own sort of detailed planning and only own sort of detailed planning and only own sort of detailed planning and only ask you help when it's needed uh it can ask you help when it's needed uh it can ask you help when it's needed uh it can use this is actually kind of interesting use this is actually kind of interesting use this is actually kind of interesting I think in the essay you said I mean I think in the essay you said I mean I think in the essay you said I mean again it's a bet that it's not going to again it's a bet that it's not going to again it's a bet that it's not going to be embodied but it can control embodied be embodied but it can control embodied be embodied but it can control embodied tools so it can control tools robots

tools so it can control tools robots tools so it can control tools robots Laboratory equipment the resource used Laboratory equipment the resource used Laboratory equipment the resource used to train it can then be repurposed to to train it can then be repurposed to to train it can then be repurposed to run millions of copies of it and each of run millions of copies of it and each of run millions of copies of it and each of those copies would be independent that those copies would be independent that those copies would be independent that can do their own independent work so you can do their own independent work so you can do their own independent work so you can do the cloning of the intelligence can do the cloning of the intelligence can do the cloning of the intelligence system yeah yeah I mean you you might system yeah yeah I mean you you might system yeah yeah I mean you you might imagine from outside the field that like imagine from outside the field that like imagine from outside the field that like there's only one of these right that there's only one of these right that there's only one of these right that like you made it you've only made one like you made it you've only made one like you made it you've only made one but the truth is that like the scale up but the truth is that like the scale up but the truth is that like the scale up is very quick like we we do this today is very quick like we we do this today is very quick like we we do this today we make a model and then we deploy we make a model and then we deploy we make a model and then we deploy thousands maybe tens of thousands of thousands maybe tens of thousands of thousands maybe tens of thousands of instances of it I think by the time you instances of it I think by the time you instances of it I think by the time you know certainly within 2 to 3 years know certainly within 2 to 3 years know certainly within 2 to 3 years whether we have these superp powerful whether we have these superp powerful whether we have these superp powerful AIS or not clusters are going to get to AIS or not clusters are going to get to AIS or not clusters are going to get to the size where where you'll be able to the size where where you'll be able to the size where where you'll be able to deploy millions of these and they'll be deploy millions of these and they'll be deploy millions of these and they'll be you know faster than humans and so if you know faster than humans and so if you know faster than humans and so if your picture is oh we'll have one and your picture is oh we'll have one and your picture is oh we'll have one and it'll take a while to make them my point it'll take a while to make them my point it'll take a while to make them my point there was no actually you have millions there was no actually you have millions there was no actually you have millions of them right away and in general they of them right away and in general they of them right away and in general they can learn and can learn and can learn and act uh 10 to 100 times faster than act uh 10 to 100 times faster than act uh 10 to 100 times faster than humans so that's a really nice humans so that's a really nice humans so that's a really nice definition of powerful AI okay so that definition of powerful AI okay so that definition of powerful AI okay so that but you also write that clearly such an but you also write that clearly such an but you also write that clearly such an entity would be cap capable of solving entity would be cap capable of solving entity would be cap capable of solving very difficult problems very fast but it very difficult problems very fast but it very difficult problems very fast but it is not trivial to figure out how fast is not trivial to figure out how fast is not trivial to figure out how fast two extreme positions both seem false to two extreme positions both seem false to two extreme positions both seem false to me so the singularity is on the one me so the singularity is on the one me so the singularity is on the one extreme and the opposite On The Other extreme and the opposite On The Other extreme and the opposite On The Other Extreme can you describe each of the Extreme can you describe each of the Extreme can you describe each of the extremes yeah why so yeah let's let's extremes yeah why so yeah let's let's extremes yeah why so yeah let's let's describe the extreme so like one one describe the extreme so like one one describe the extreme so like one one extreme would be well look um you know extreme would be well look um you know extreme would be well look um you know uh if we look at kind of evolutionary uh if we look at kind of evolutionary uh if we look at kind of evolutionary history like there was this big history like there was this big history like there was this big acceleration where you know for hundreds acceleration where you know for hundreds acceleration where you know for hundreds of thousands of years we just had like

of thousands of years we just had like of thousands of years we just had like you know single cell organisms and then you know single cell organisms and then you know single cell organisms and then we had mammals and then we had apes and we had mammals and then we had apes and we had mammals and then we had apes and then that quickly turned to humans then that quickly turned to humans then that quickly turned to humans humans quickly built industrial humans quickly built industrial humans quickly built industrial civilization and so this is going to civilization and so this is going to civilization and so this is going to keep speeding up and there's no cealing keep speeding up and there's no cealing keep speeding up and there's no cealing at the human level once models get much at the human level once models get much at the human level once models get much much smarter than humans they'll get much smarter than humans they'll get much smarter than humans they'll get really good at building the next models really good at building the next models really good at building the next models and you know if you write down like a and you know if you write down like a and you know if you write down like a simple differential equation like this simple differential equation like this simple differential equation like this is an exponential and so what's what's is an exponential and so what's what's is an exponential and so what's what's going to happen is that uh models will going to happen is that uh models will going to happen is that uh models will build faster models models will build build faster models models will build build faster models models will build faster models and those models will faster models and those models will faster models and those models will build you know Nano that can like take build you know Nano that can like take build you know Nano that can like take over the world and produce much more over the world and produce much more over the world and produce much more energy than you could produce otherwise energy than you could produce otherwise energy than you could produce otherwise and and so if you just kind of like and and so if you just kind of like and and so if you just kind of like solve this abstract differential solve this abstract differential solve this abstract differential equation then like 5 days after we you equation then like 5 days after we you equation then like 5 days after we you know we build the first AI That's more know we build the first AI That's more know we build the first AI That's more powerful than humans then then uh you powerful than humans then then uh you powerful than humans then then uh you know like the world will be filled with know like the world will be filled with know like the world will be filled with these AIS and every possible technology these AIS and every possible technology these AIS and every possible technology that could be invented like will be that could be invented like will be that could be invented like will be invented um I'm caricaturing this a invented um I'm caricaturing this a invented um I'm caricaturing this a little bit um uh but I you know I think little bit um uh but I you know I think little bit um uh but I you know I think that's one extreme and the reason that I that's one extreme and the reason that I that's one extreme and the reason that I think that's not the case is is that one think that's not the case is is that one think that's not the case is is that one I think they just neglect like the laws I think they just neglect like the laws I think they just neglect like the laws of physics like it's only possible to do of physics like it's only possible to do of physics like it's only possible to do things so fast in the physical world things so fast in the physical world things so fast in the physical world like some of those Loops go through you like some of those Loops go through you like some of those Loops go through you know producing faster Hardware um uh know producing faster Hardware um uh know producing faster Hardware um uh takes a long time to produce faster takes a long time to produce faster takes a long time to produce faster Hardware things take a long time there's Hardware things take a long time there's Hardware things take a long time there's this issue of complexity like I think no this issue of complexity like I think no this issue of complexity like I think no matter how smart you are like you know matter how smart you are like you know matter how smart you are like you know people talk about oh we can make models people talk about oh we can make models people talk about oh we can make models the biological systems it'll do the biological systems it'll do the biological systems it'll do everything the biological systems look I everything the biological systems look I everything the biological systems look I think computational modeling can do a think computational modeling can do a think computational modeling can do a lot I did a lot of computational lot I did a lot of computational lot I did a lot of computational modeling when I worked in biology but

modeling when I worked in biology but modeling when I worked in biology but like like like just there are a lot of things that you just there are a lot of things that you just there are a lot of things that you can't predict how they're you know can't predict how they're you know can't predict how they're you know they're they're complex enough that like they're they're complex enough that like they're they're complex enough that like just iterating just running the just iterating just running the just iterating just running the experiment is going to beat any modeling experiment is going to beat any modeling experiment is going to beat any modeling no matter how smart the system doing the no matter how smart the system doing the no matter how smart the system doing the modeling is oh even if it's not modeling is oh even if it's not modeling is oh even if it's not interacting with the physical world just interacting with the physical world just interacting with the physical world just the modeling is going to be hard yeah I the modeling is going to be hard yeah I the modeling is going to be hard yeah I think well the modeling is going to be think well the modeling is going to be think well the modeling is going to be hard and getting the model to to to to hard and getting the model to to to to hard and getting the model to to to to match the physical world is going to be match the physical world is going to be match the physical world is going to be all right so he does have to intera the all right so he does have to intera the all right so he does have to intera the physical world to verify but it's just physical world to verify but it's just physical world to verify but it's just you know you just look at even the you know you just look at even the you know you just look at even the simplest problems like I you know I simplest problems like I you know I simplest problems like I you know I think I talk about like you know the think I talk about like you know the think I talk about like you know the three body problem or simple chaotic three body problem or simple chaotic three body problem or simple chaotic prediction like you know or or like prediction like you know or or like prediction like you know or or like predicting the economy it's really hard predicting the economy it's really hard predicting the economy it's really hard to predict the economy two years out to predict the economy two years out to predict the economy two years out like maybe the case is like you know like maybe the case is like you know like maybe the case is like you know normal you know humans can predict normal you know humans can predict normal you know humans can predict what's going to happen in the economy in what's going to happen in the economy in what's going to happen in the economy in the next quarter although they can't the next quarter although they can't the next quarter although they can't really do that maybe a maybe a AI system really do that maybe a maybe a AI system really do that maybe a maybe a AI system that's you know a zillion times smarter that's you know a zillion times smarter that's you know a zillion times smarter can only predict it out a year or can only predict it out a year or can only predict it out a year or something instead of instead of a you something instead of instead of a you something instead of instead of a you know you have the these kind of know you have the these kind of know you have the these kind of exponential increase in computer exponential increase in computer exponential increase in computer intelligence for linear increase in in intelligence for linear increase in in intelligence for linear increase in in in ability to predict same with again in ability to predict same with again in ability to predict same with again like you know biological molecules like you know biological molecules like you know biological molecules molecules interacting you don't know molecules interacting you don't know molecules interacting you don't know what's going to happen when you perturb what's going to happen when you perturb what's going to happen when you perturb a when you perturb a complex system you a when you perturb a complex system you a when you perturb a complex system you can find simple Parts in it if you're can find simple Parts in it if you're can find simple Parts in it if you're smarter you're better at finding these smarter you're better at finding these smarter you're better at finding these simple parts and then I think human simple parts and then I think human simple parts and then I think human institutions human institutions are just institutions human institutions are just institutions human institutions are just are are really difficult like it's you are are really difficult like it's you are are really difficult like it's you know it's it's been hard to get people I know it's it's been hard to get people I know it's it's been hard to get people I won't give specific examples but it's

won't give specific examples but it's won't give specific examples but it's been hard to get people to adopt even been hard to get people to adopt even been hard to get people to adopt even the technologies that we've developed the technologies that we've developed the technologies that we've developed even ones where the case for their even ones where the case for their even ones where the case for their efficacy is very very strong um you know efficacy is very very strong um you know efficacy is very very strong um you know people have concerns they think things people have concerns they think things people have concerns they think things are conspiracy theories like it's it's are conspiracy theories like it's it's are conspiracy theories like it's it's just been it's been very difficult it's just been it's been very difficult it's just been it's been very difficult it's also been very difficult to get you know also been very difficult to get you know also been very difficult to get you know very simple things through the very simple things through the very simple things through the regulatory system right I think you know regulatory system right I think you know regulatory system right I think you know and you know I I don't want to just and you know I I don't want to just and you know I I don't want to just spage anyone who you know you know work spage anyone who you know you know work spage anyone who you know you know work Works in regulator regulatory systems of Works in regulator regulatory systems of Works in regulator regulatory systems of any technology there are hard trade-offs any technology there are hard trade-offs any technology there are hard trade-offs they have to deal with they have to save they have to deal with they have to save they have to deal with they have to save lives but but the system as a whole I lives but but the system as a whole I lives but but the system as a whole I think makes some obvious tradeoffs that think makes some obvious tradeoffs that think makes some obvious tradeoffs that are very far from maximizing human are very far from maximizing human are very far from maximizing human welfare and so if we bring AI systems welfare and so if we bring AI systems welfare and so if we bring AI systems into this you into this you into this you know into these human systems often the know into these human systems often the know into these human systems often the level of intelligence may just not be level of intelligence may just not be level of intelligence may just not be the limiting factor right it it it just the limiting factor right it it it just the limiting factor right it it it just may be that it takes a long time to do may be that it takes a long time to do may be that it takes a long time to do something now if the AI system uh something now if the AI system uh something now if the AI system uh circumvented all governments if it just circumvented all governments if it just circumvented all governments if it just said I'm dictator of the world and I'm said I'm dictator of the world and I'm said I'm dictator of the world and I'm going to do whatever some of these going to do whatever some of these going to do whatever some of these things it could do again the things things it could do again the things things it could do again the things having to do with complexity I I I still having to do with complexity I I I still having to do with complexity I I I still think a lot of things would take a while think a lot of things would take a while think a lot of things would take a while I don't think it helps that the AI I don't think it helps that the AI I don't think it helps that the AI systems can produce a lot of energy or systems can produce a lot of energy or systems can produce a lot of energy or go to the moon like some people in go to the moon like some people in go to the moon like some people in comments responded to the essay saying comments responded to the essay saying comments responded to the essay saying the AI system can produce a lot of the AI system can produce a lot of the AI system can produce a lot of energy and smarter AI systems that's energy and smarter AI systems that's energy and smarter AI systems that's missing the point that kind of cycle missing the point that kind of cycle missing the point that kind of cycle doesn't solve the key problems that I'm doesn't solve the key problems that I'm doesn't solve the key problems that I'm talking about here um so I think I think

talking about here um so I think I think talking about here um so I think I think a bunch of people missed the point there a bunch of people missed the point there a bunch of people missed the point there but even if it were completely on but even if it were completely on but even if it were completely on aligned and you know could get around aligned and you know could get around aligned and you know could get around all these human obstacles it would have all these human obstacles it would have all these human obstacles it would have trouble but again if you want this to be trouble but again if you want this to be trouble but again if you want this to be an AI system that doesn't take over the an AI system that doesn't take over the an AI system that doesn't take over the world that doesn't destroy Humanity then world that doesn't destroy Humanity then world that doesn't destroy Humanity then then basically you know it's it's it's then basically you know it's it's it's then basically you know it's it's it's going to need to follow basic human laws going to need to follow basic human laws going to need to follow basic human laws right where you know if if we want to right where you know if if we want to right where you know if if we want to have an actually good world like we're have an actually good world like we're have an actually good world like we're going to have to have an AI system that going to have to have an AI system that going to have to have an AI system that that interacts with humans not one that that interacts with humans not one that that interacts with humans not one that kind of creates its own legal system or kind of creates its own legal system or kind of creates its own legal system or disregards all the laws or all of that disregards all the laws or all of that disregards all the laws or all of that so as inefficient as these processes are so as inefficient as these processes are so as inefficient as these processes are you know we're going to have to deal you know we're going to have to deal you know we're going to have to deal with them because there there needs to with them because there there needs to with them because there there needs to be some popular and Democratic be some popular and Democratic be some popular and Democratic legitimacy in how these systems are legitimacy in how these systems are legitimacy in how these systems are rolled out we can't have a small group rolled out we can't have a small group rolled out we can't have a small group of people who are developing these of people who are developing these of people who are developing these systems say this is what's best for systems say this is what's best for systems say this is what's best for everyone right I think it's wrong and I everyone right I think it's wrong and I everyone right I think it's wrong and I think in practice is not going to work think in practice is not going to work think in practice is not going to work anyway so you put all those things anyway so you put all those things anyway so you put all those things together and you know we're not we're together and you know we're not we're together and you know we're not we're not g to we're not going to you know not g to we're not going to you know not g to we're not going to you know change the world and upload everyone in change the world and upload everyone in change the world and upload everyone in five minutes uh I I I just I don't think five minutes uh I I I just I don't think five minutes uh I I I just I don't think it I A A I don't think it's going to it I A A I don't think it's going to it I A A I don't think it's going to happen and be to some in you know to the happen and be to some in you know to the happen and be to some in you know to the extent that it could happen it's it's extent that it could happen it's it's extent that it could happen it's it's not the way to lead to a good world so not the way to lead to a good world so not the way to lead to a good world so that's on one side on the other side that's on one side on the other side that's on one side on the other side there's another set of perspectives there's another set of perspectives there's another set of perspectives which I have actually in some ways more which I have actually in some ways more which I have actually in some ways more sympathy for which is look we've seen sympathy for which is look we've seen sympathy for which is look we've seen big productivity increases before right big productivity increases before right big productivity increases before right you know economists are familiar with you know economists are familiar with you know economists are familiar with studying the productivity increases that studying the productivity increases that studying the productivity increases that came from the computer Revolution and came from the computer Revolution and came from the computer Revolution and internet Revolution and generally those internet Revolution and generally those internet Revolution and generally those productivity increases were

productivity increases were productivity increases were underwhelming they were less than you underwhelming they were less than you underwhelming they were less than you than you might imagine um there was a than you might imagine um there was a than you might imagine um there was a quote from Robert solo you see the quote from Robert solo you see the quote from Robert solo you see the computer Revolution everywhere except computer Revolution everywhere except computer Revolution everywhere except the productivity statistics so why is the productivity statistics so why is the productivity statistics so why is this the case people point to the this the case people point to the this the case people point to the structure of firms the structure of structure of firms the structure of structure of firms the structure of Enterprises how um uh you know how slow Enterprises how um uh you know how slow Enterprises how um uh you know how slow it's been to roll out our existing it's been to roll out our existing it's been to roll out our existing technology to very poor parts of the technology to very poor parts of the technology to very poor parts of the world which I talk about in the essay world which I talk about in the essay world which I talk about in the essay right how do we get these Technologies right how do we get these Technologies right how do we get these Technologies to the poorest parts of the world that to the poorest parts of the world that to the poorest parts of the world that are behind on cell phone technology are behind on cell phone technology are behind on cell phone technology computers medicine let alone you know computers medicine let alone you know computers medicine let alone you know new fangled AI that hasn't been invented new fangled AI that hasn't been invented new fangled AI that hasn't been invented yet um so you could have a perspective yet um so you could have a perspective yet um so you could have a perspective that's like well this is amazing that's like well this is amazing that's like well this is amazing technically but it's all a nothing burer technically but it's all a nothing burer technically but it's all a nothing burer um uh you know I think um Tyler Cowan um uh you know I think um Tyler Cowan um uh you know I think um Tyler Cowan who who wrote something response to my who who wrote something response to my who who wrote something response to my essay has that perspective I think he essay has that perspective I think he essay has that perspective I think he thinks the radical change will happen thinks the radical change will happen thinks the radical change will happen eventually but he thinks it'll take 50 eventually but he thinks it'll take 50 eventually but he thinks it'll take 50 or 100 years and and you could have even or 100 years and and you could have even or 100 years and and you could have even more static perspectives on the whole more static perspectives on the whole more static perspectives on the whole thing I think there's some truth to it I thing I think there's some truth to it I thing I think there's some truth to it I think the time scale is just is just too think the time scale is just is just too think the time scale is just is just too long um and and I can see it I can long um and and I can see it I can long um and and I can see it I can actually see both sides with today's AI actually see both sides with today's AI actually see both sides with today's AI so uh you know a lot of our customers so uh you know a lot of our customers so uh you know a lot of our customers are large Enterprises who are used to are large Enterprises who are used to are large Enterprises who are used to doing things a certain way um I've also doing things a certain way um I've also doing things a certain way um I've also seen it in talking to governments right seen it in talking to governments right seen it in talking to governments right those are those are prototypical you those are those are prototypical you those are those are prototypical you know institutions entities that are slow know institutions entities that are slow know institutions entities that are slow to change uh but the dynamic I see over to change uh but the dynamic I see over to change uh but the dynamic I see over and over again is yes it takes a long and over again is yes it takes a long and over again is yes it takes a long time to move the ship yes there's a lot time to move the ship yes there's a lot time to move the ship yes there's a lot of resistance and lack of understanding

of resistance and lack of understanding of resistance and lack of understanding but the thing that makes me feel that but the thing that makes me feel that but the thing that makes me feel that progress will in the end happen progress will in the end happen progress will in the end happen moderately fast not incredibly fast but moderately fast not incredibly fast but moderately fast not incredibly fast but moderately fast is that you talk to what moderately fast is that you talk to what moderately fast is that you talk to what I find is I find over and over again I find is I find over and over again I find is I find over and over again again in large companies even in again in large companies even in again in large companies even in governments um which have been actually governments um which have been actually governments um which have been actually surprisingly forward leaning uh you find surprisingly forward leaning uh you find surprisingly forward leaning uh you find two things that move things forward one two things that move things forward one two things that move things forward one you find a small fraction of people you find a small fraction of people you find a small fraction of people within a company within a government who within a company within a government who within a company within a government who really see the big picture who see the really see the big picture who see the really see the big picture who see the whole scaling hypothesis who understand whole scaling hypothesis who understand whole scaling hypothesis who understand where AI is going or at least understand where AI is going or at least understand where AI is going or at least understand where it's going within their industry where it's going within their industry where it's going within their industry and there are a few people like that and there are a few people like that and there are a few people like that within the current within the current US within the current within the current US within the current within the current US government who really see the whole government who really see the whole government who really see the whole picture and and those people see that picture and and those people see that picture and and those people see that this is the most important thing in the this is the most important thing in the this is the most important thing in the world until they agitate for it and the world until they agitate for it and the world until they agitate for it and the thing they they alone are not enough to thing they they alone are not enough to thing they they alone are not enough to succeed because they are a small set of succeed because they are a small set of succeed because they are a small set of people within a large organization people within a large organization people within a large organization but as the technology starts to roll out but as the technology starts to roll out but as the technology starts to roll out as it succeeds in some places in the as it succeeds in some places in the as it succeeds in some places in the folks who are most willing to adopt it folks who are most willing to adopt it folks who are most willing to adopt it the Spectre of competition gives them a the Spectre of competition gives them a the Spectre of competition gives them a wind at their backs because they can wind at their backs because they can wind at their backs because they can point within their large organization point within their large organization point within their large organization they can say look these other guys are they can say look these other guys are they can say look these other guys are doing this right you know One bank can doing this right you know One bank can doing this right you know One bank can say look this new fangled hedge fund is say look this new fangled hedge fund is say look this new fangled hedge fund is doing this thing they're going to eat doing this thing they're going to eat doing this thing they're going to eat our lunch in the US we can say we're our lunch in the US we can say we're our lunch in the US we can say we're afraid China's going to get there before afraid China's going to get there before afraid China's going to get there before before we are uh and that combination before we are uh and that combination before we are uh and that combination the Spectre of competition plus a few the Spectre of competition plus a few the Spectre of competition plus a few Visionaries Within These you know within Visionaries Within These you know within Visionaries Within These you know within these the organizations that in many

these the organizations that in many these the organizations that in many ways are are sclerotic you put those two ways are are sclerotic you put those two ways are are sclerotic you put those two things together and it actually makes things together and it actually makes things together and it actually makes something happen I mean it's interesting something happen I mean it's interesting something happen I mean it's interesting it's a balanced fight between the two it's a balanced fight between the two it's a balanced fight between the two because inertia is very powerful but but because inertia is very powerful but but because inertia is very powerful but but but eventually over enough time the but eventually over enough time the but eventually over enough time the Innovative approach breaks through um Innovative approach breaks through um Innovative approach breaks through um and I've seen that happen I've seen the and I've seen that happen I've seen the and I've seen that happen I've seen the Arc of that over and over again and it's Arc of that over and over again and it's Arc of that over and over again and it's like the the barriers are there the the like the the barriers are there the the like the the barriers are there the the barriers to progress the complexity not barriers to progress the complexity not barriers to progress the complexity not knowing how to use the model or how to knowing how to use the model or how to knowing how to use the model or how to deploy them are there and and for a bit deploy them are there and and for a bit deploy them are there and and for a bit it seems like they're going to last it seems like they're going to last it seems like they're going to last forever like change doesn't happen but forever like change doesn't happen but forever like change doesn't happen but then eventually change happens and then eventually change happens and then eventually change happens and always comes from a few people I felt always comes from a few people I felt always comes from a few people I felt the same way when I was an advocate of the same way when I was an advocate of the same way when I was an advocate of the scaling hypothesis within the AI the scaling hypothesis within the AI the scaling hypothesis within the AI field itself and others didn't get it it field itself and others didn't get it it field itself and others didn't get it it felt like no one would ever get it it felt like no one would ever get it it felt like no one would ever get it it felt like then it felt like we had a felt like then it felt like we had a felt like then it felt like we had a secret almost no one ever had and then a secret almost no one ever had and then a secret almost no one ever had and then a couple years later everyone has the couple years later everyone has the couple years later everyone has the secret and so I think that's how it's secret and so I think that's how it's secret and so I think that's how it's going to go with deployment to AI in the going to go with deployment to AI in the going to go with deployment to AI in the world it's going to the the barriers are world it's going to the the barriers are world it's going to the the barriers are going to fall apart gradually and then going to fall apart gradually and then going to fall apart gradually and then all at once and so I think this is going all at once and so I think this is going all at once and so I think this is going to be more and this is just an instinct to be more and this is just an instinct to be more and this is just an instinct I could I could easily see how I'm wrong I could I could easily see how I'm wrong I could I could easily see how I'm wrong I think it's going to be more like 10 I think it's going to be more like 10 I think it's going to be more like 10 five or 10 years as I say in the essay five or 10 years as I say in the essay five or 10 years as I say in the essay then it's going to be 50 or 100 years I then it's going to be 50 or 100 years I then it's going to be 50 or 100 years I also think it's going to be five or 10 also think it's going to be five or 10 also think it's going to be five or 10 years years years more than it's going to be you know five more than it's going to be you know five more than it's going to be you know five or 10 hours uh uh because I've just I've or 10 hours uh uh because I've just I've or 10 hours uh uh because I've just I've just seen how human systems work and I just seen how human systems work and I just seen how human systems work and I think a lot of these people who write

think a lot of these people who write think a lot of these people who write down the differential equations who say down the differential equations who say down the differential equations who say AI is going to make more powerful AI who AI is going to make more powerful AI who AI is going to make more powerful AI who can't understand how it could possibly can't understand how it could possibly can't understand how it could possibly be the case that these things won't be the case that these things won't be the case that these things won't won't change so fast I think they don't won't change so fast I think they don't won't change so fast I think they don't understand these things so what to use understand these things so what to use understand these things so what to use the timeline to where we achieve the timeline to where we achieve the timeline to where we achieve AGI AKA powerful AI AKA super useful AI AGI AKA powerful AI AKA super useful AI AGI AKA powerful AI AKA super useful AI I'm start calling it that it's a debate I'm start calling it that it's a debate I'm start calling it that it's a debate it's a debate about it's a debate about it's a debate about naming um you know unpure intelligence naming um you know unpure intelligence naming um you know unpure intelligence you can smarter than a Nobel Prize you can smarter than a Nobel Prize you can smarter than a Nobel Prize winner in every relevant discipline and winner in every relevant discipline and winner in every relevant discipline and all the things we've said modality you all the things we've said modality you all the things we've said modality you can go and do stuff on its own for days can go and do stuff on its own for days can go and do stuff on its own for days weeks and do biology experiments uh on weeks and do biology experiments uh on weeks and do biology experiments uh on its own in one you know what let's just its own in one you know what let's just its own in one you know what let's just stick to biology because yeah I you you stick to biology because yeah I you you stick to biology because yeah I you you sold me on the whole biology and health sold me on the whole biology and health sold me on the whole biology and health section That's so exciting from um from section That's so exciting from um from section That's so exciting from um from a just I was getting giddy from a a just I was getting giddy from a a just I was getting giddy from a scientific perspective it made me want scientific perspective it made me want scientific perspective it made me want to be a biologist it's almost it's it's to be a biologist it's almost it's it's to be a biologist it's almost it's it's so no no that this was the feeling I had so no no that this was the feeling I had so no no that this was the feeling I had when I was writing it that it's it's when I was writing it that it's it's when I was writing it that it's it's like this would be such a beautiful like this would be such a beautiful like this would be such a beautiful future if we can if we can just if we future if we can if we can just if we future if we can if we can just if we can just make it happen right if we can can just make it happen right if we can can just make it happen right if we can just get the get the landmines out of just get the get the landmines out of just get the get the landmines out of the way and and and and make it happen the way and and and and make it happen the way and and and and make it happen there's there's so much there's so much there's there's so much there's so much there's there's so much there's so much Beauty and and and and and elegance and Beauty and and and and and elegance and Beauty and and and and and elegance and moral force behind it if if we can if we moral force behind it if if we can if we moral force behind it if if we can if we can just and it's something we should can just and it's something we should can just and it's something we should all be able to agree on right like as all be able to agree on right like as all be able to agree on right like as much as we fight about about all these much as we fight about about all these much as we fight about about all these political questions is is this something political questions is is this something political questions is is this something that could actually bring us together um

that could actually bring us together um that could actually bring us together um but you were asking when when will we but you were asking when when will we but you were asking when when will we get this when when do you think what's get this when when do you think what's get this when when do you think what's just put numbers on so you know this just put numbers on so you know this just put numbers on so you know this this is of course the thing I've been this is of course the thing I've been this is of course the thing I've been grappling with for many years and I'm grappling with for many years and I'm grappling with for many years and I'm not I'm not at all confident every time not I'm not at all confident every time not I'm not at all confident every time if I say 2026 or 2027 there will be like if I say 2026 or 2027 there will be like if I say 2026 or 2027 there will be like a zillion like people on Twitter who a zillion like people on Twitter who a zillion like people on Twitter who will be like he icoo said 2026 2020 and will be like he icoo said 2026 2020 and will be like he icoo said 2026 2020 and it'll be repeated for like the next two it'll be repeated for like the next two it'll be repeated for like the next two years that like this is definitely when years that like this is definitely when years that like this is definitely when I think it's going to happen um so who I think it's going to happen um so who I think it's going to happen um so who whoever's exerting these clips will will whoever's exerting these clips will will whoever's exerting these clips will will we we'll we'll crop out the thing I just we we'll we'll crop out the thing I just we we'll we'll crop out the thing I just said and and only say the thing I'm said and and only say the thing I'm said and and only say the thing I'm about to say um but I'll just say it about to say um but I'll just say it about to say um but I'll just say it anyway um have so so uh if you anyway um have so so uh if you anyway um have so so uh if you extrapolate the curves that we've had so extrapolate the curves that we've had so extrapolate the curves that we've had so far right if if you say well I don't far right if if you say well I don't far right if if you say well I don't know we're starting to get to like PhD know we're starting to get to like PhD know we're starting to get to like PhD level and and last year we were at um uh level and and last year we were at um uh level and and last year we were at um uh undergraduate level in the year before undergraduate level in the year before undergraduate level in the year before we were at like the level of a high we were at like the level of a high we were at like the level of a high school student again you can you can school student again you can you can school student again you can you can quibble with at what tasks and for what quibble with at what tasks and for what quibble with at what tasks and for what we're still missing modalities but those we're still missing modalities but those we're still missing modalities but those are being added like computer use was are being added like computer use was are being added like computer use was added like image in was added like image added like image in was added like image added like image in was added like image generation has been added if you just generation has been added if you just generation has been added if you just kind of like and this is totally kind of like and this is totally kind of like and this is totally unscientific but if you just kind of unscientific but if you just kind of unscientific but if you just kind of like eyeball the rate at which these like eyeball the rate at which these like eyeball the rate at which these capabilities are increasing it does make capabilities are increasing it does make capabilities are increasing it does make you think that we'll get there by 2026 you think that we'll get there by 2026 you think that we'll get there by 2026 or 2027 again lots of things could or 2027 again lots of things could or 2027 again lots of things could derail it we could run out of data you derail it we could run out of data you derail it we could run out of data you know we might not be able to scale know we might not be able to scale know we might not be able to scale clusters as much as we want like you clusters as much as we want like you clusters as much as we want like you know maybe Taiwan gets blown up or know maybe Taiwan gets blown up or know maybe Taiwan gets blown up or something and you know then we can't something and you know then we can't something and you know then we can't produce as many gpus as we want so there

produce as many gpus as we want so there produce as many gpus as we want so there there are all kinds of things that could there are all kinds of things that could there are all kinds of things that could could derail the whole process so I could derail the whole process so I could derail the whole process so I don't fully believe the straight line don't fully believe the straight line don't fully believe the straight line extrapolation but if you believe the extrapolation but if you believe the extrapolation but if you believe the straight line extrapolation you'll you straight line extrapolation you'll you straight line extrapolation you'll you we'll get there in 2026 or 2027 I think we'll get there in 2026 or 2027 I think we'll get there in 2026 or 2027 I think the most likely is that there's some the most likely is that there's some the most likely is that there's some mild delay relative to that um mild delay relative to that um mild delay relative to that um I don't know what that delay is but I I don't know what that delay is but I I don't know what that delay is but I think it could happen on schedule I think it could happen on schedule I think it could happen on schedule I think there could be a mild delay I think there could be a mild delay I think there could be a mild delay I think there are still worlds where it think there are still worlds where it think there are still worlds where it doesn't happen in in a hundred years doesn't happen in in a hundred years doesn't happen in in a hundred years those world the number of those worlds those world the number of those worlds those world the number of those worlds is rapidly decreasing we are rapidly is rapidly decreasing we are rapidly is rapidly decreasing we are rapidly running out of truly convincing Brockers running out of truly convincing Brockers running out of truly convincing Brockers truly compelling reasons why this will truly compelling reasons why this will truly compelling reasons why this will not happen in the next few years there not happen in the next few years there not happen in the next few years there were a lot more in 2020 um although my were a lot more in 2020 um although my were a lot more in 2020 um although my my guest my hunch at that time was that my guest my hunch at that time was that my guest my hunch at that time was that we will make it through all those we will make it through all those we will make it through all those blockers so sitting as someone who has blockers so sitting as someone who has blockers so sitting as someone who has seen most of the blockers cleared out of seen most of the blockers cleared out of seen most of the blockers cleared out of the way I kind of suspect my hunch my the way I kind of suspect my hunch my the way I kind of suspect my hunch my suspicion is that the rest of them will suspicion is that the rest of them will suspicion is that the rest of them will not block us uh but you know look look not block us uh but you know look look not block us uh but you know look look at look at the end of the day like I at look at the end of the day like I at look at the end of the day like I don't want to represent this as a don't want to represent this as a don't want to represent this as a scientific prediction people call them scientific prediction people call them scientific prediction people call them scaling laws that's a misnomer like Mo's scaling laws that's a misnomer like Mo's scaling laws that's a misnomer like Mo's law is is is a misnomer Moors laws law is is is a misnomer Moors laws law is is is a misnomer Moors laws scaling laws they're not laws of the scaling laws they're not laws of the scaling laws they're not laws of the universe they're empirical regularities universe they're empirical regularities universe they're empirical regularities I am going to bet in favor of them I am going to bet in favor of them I am going to bet in favor of them continuing but I'm not certain of that continuing but I'm not certain of that continuing but I'm not certain of that so you extensively describe sort of the so you extensively describe sort of the so you extensively describe sort of the compressed 21st century how AGI will compressed 21st century how AGI will compressed 21st century how AGI will help help help uh set forth a chain of breakthroughs in uh set forth a chain of breakthroughs in uh set forth a chain of breakthroughs in biology and medicine that help us in all biology and medicine that help us in all biology and medicine that help us in all these kinds of ways that I mentioned so these kinds of ways that I mentioned so these kinds of ways that I mentioned so how do you think what are the early how do you think what are the early how do you think what are the early steps it might do and by the way I asked

steps it might do and by the way I asked steps it might do and by the way I asked Claude good questions to ask Claude good questions to ask Claude good questions to ask you and Claude told me uh to ask what do you and Claude told me uh to ask what do you and Claude told me uh to ask what do you think is a typical day for a you think is a typical day for a you think is a typical day for a biologist working on AGI look like under biologist working on AGI look like under biologist working on AGI look like under in this future yeah yeah Claud is in this future yeah yeah Claud is in this future yeah yeah Claud is curious let me well let me start with curious let me well let me start with curious let me well let me start with your first questions and then I'll then your first questions and then I'll then your first questions and then I'll then I'll answer that Claude Claude wants to I'll answer that Claude Claude wants to I'll answer that Claude Claude wants to know what's in his future right exactly know what's in his future right exactly know what's in his future right exactly who's it who am I going to be working who's it who am I going to be working who's it who am I going to be working with exactly um so I think one of the with exactly um so I think one of the with exactly um so I think one of the things I went hard on in when I went things I went hard on in when I went things I went hard on in when I went hard on in the essay is let me go back hard on in the essay is let me go back hard on in the essay is let me go back to this idea of because it's it's really to this idea of because it's it's really to this idea of because it's it's really had had an you know had an impact on me had had an you know had an impact on me had had an you know had an impact on me this idea that within large this idea that within large this idea that within large organizations and systems there end up organizations and systems there end up organizations and systems there end up being a few people or a few new ideas being a few people or a few new ideas being a few people or a few new ideas who kind of cause things to go in a who kind of cause things to go in a who kind of cause things to go in a different direction they would have different direction they would have different direction they would have before who who kind of a before who who kind of a before who who kind of a disproportionately affect the the disproportionately affect the the disproportionately affect the the trajectory there's a bunch of kind of trajectory there's a bunch of kind of trajectory there's a bunch of kind of the same thing going on right if you the same thing going on right if you the same thing going on right if you think about the health world there's think about the health world there's think about the health world there's like you know trillions of dollars to like you know trillions of dollars to like you know trillions of dollars to pay out Medicare and you know other pay out Medicare and you know other pay out Medicare and you know other health insurance and then the NIH is is health insurance and then the NIH is is health insurance and then the NIH is is 100 billion and then if I think of like 100 billion and then if I think of like 100 billion and then if I think of like the the few things that have really the the few things that have really the the few things that have really revolutionized anything it could be revolutionized anything it could be revolutionized anything it could be encapsulated in a small small fraction encapsulated in a small small fraction encapsulated in a small small fraction of that and so when I think of like of that and so when I think of like of that and so when I think of like where will AI have an impact I'm like where will AI have an impact I'm like where will AI have an impact I'm like can AI turn that small fraction into a can AI turn that small fraction into a can AI turn that small fraction into a much larger fraction and raise its much larger fraction and raise its much larger fraction and raise its quality and within biology my experience quality and within biology my experience quality and within biology my experience within biology is that the biggest within biology is that the biggest within biology is that the biggest problem of biology is that you can't see problem of biology is that you can't see problem of biology is that you can't see what's going on you you have very little what's going on you you have very little what's going on you you have very little ability to see what's going on and even ability to see what's going on and even ability to see what's going on and even less ability to change it right what you

less ability to change it right what you less ability to change it right what you have is this like like from this you have is this like like from this you have is this like like from this you have to infer that there's a bunch of have to infer that there's a bunch of have to infer that there's a bunch of cells that within each cell is you know cells that within each cell is you know cells that within each cell is you know uh uh three billion base pairs of DNA uh uh three billion base pairs of DNA uh uh three billion base pairs of DNA built according to a genetic code uh uh built according to a genetic code uh uh built according to a genetic code uh uh and you know there are all these and you know there are all these and you know there are all these processes that are just going on without processes that are just going on without processes that are just going on without any ability of us as you know un any ability of us as you know un any ability of us as you know un augmented humans to affect it these augmented humans to affect it these augmented humans to affect it these cells are dividing most of the time cells are dividing most of the time cells are dividing most of the time that's healthy but sometimes that that's healthy but sometimes that that's healthy but sometimes that process goes wrong and that's cancer um process goes wrong and that's cancer um process goes wrong and that's cancer um the cells are aging your skin may change the cells are aging your skin may change the cells are aging your skin may change color develops wrinkles as you as you color develops wrinkles as you as you color develops wrinkles as you as you age and all of this is determined by age and all of this is determined by age and all of this is determined by these processes all these proteins being these processes all these proteins being these processes all these proteins being produced transported to various parts of produced transported to various parts of produced transported to various parts of the cells binding to each other and and the cells binding to each other and and the cells binding to each other and and in our initial State about biology we in our initial State about biology we in our initial State about biology we didn't even know that these cells didn't even know that these cells didn't even know that these cells existed we had to invent microscopes to existed we had to invent microscopes to existed we had to invent microscopes to observe the cells we had to uh we had to observe the cells we had to uh we had to observe the cells we had to uh we had to invent more powerful microscopes to see invent more powerful microscopes to see invent more powerful microscopes to see you know below the level of the cell to you know below the level of the cell to you know below the level of the cell to the level of molecules we had to invent the level of molecules we had to invent the level of molecules we had to invent x-ray crystallography to see the DNA we x-ray crystallography to see the DNA we x-ray crystallography to see the DNA we had to invent Gene sequencing to read had to invent Gene sequencing to read had to invent Gene sequencing to read the DNA now you know we had to invent the DNA now you know we had to invent the DNA now you know we had to invent protein folding technology to you know protein folding technology to you know protein folding technology to you know to predict how it would fold and how to predict how it would fold and how to predict how it would fold and how they bind and how these things bind to they bind and how these things bind to they bind and how these things bind to each other uh you know we had to we had each other uh you know we had to we had each other uh you know we had to we had to invent various techniques for now we to invent various techniques for now we to invent various techniques for now we can edit the G the DNA as of you know can edit the G the DNA as of you know can edit the G the DNA as of you know with chrisopher as of the last uh uh 12 with chrisopher as of the last uh uh 12 with chrisopher as of the last uh uh 12 years so the the whole history of years so the the whole history of years so the the whole history of biology a whole big part of the history biology a whole big part of the history biology a whole big part of the history is is basically our our our our ability

is is basically our our our our ability is is basically our our our our ability to read and understand what's going on to read and understand what's going on to read and understand what's going on and our ability to reach in and and our ability to reach in and and our ability to reach in and selectively change things um and and my selectively change things um and and my selectively change things um and and my view is that there's so much more we can view is that there's so much more we can view is that there's so much more we can still do there right you can do crisper still do there right you can do crisper still do there right you can do crisper but you can do it for your whole body um but you can do it for your whole body um but you can do it for your whole body um let's say I want to do it for one let's say I want to do it for one let's say I want to do it for one particular type of cell and I want the particular type of cell and I want the particular type of cell and I want the rate of targeting the wrong cell to be rate of targeting the wrong cell to be rate of targeting the wrong cell to be very low that's still a challenge that's very low that's still a challenge that's very low that's still a challenge that's still things people are working on still things people are working on still things people are working on that's what we might need for gene that's what we might need for gene that's what we might need for gene therapy for certain diseases and so the therapy for certain diseases and so the therapy for certain diseases and so the reason I'm saying all of this and it reason I'm saying all of this and it reason I'm saying all of this and it goes beyond you know beyond this to you goes beyond you know beyond this to you goes beyond you know beyond this to you know to Gene sequencing to new types of know to Gene sequencing to new types of know to Gene sequencing to new types of nanomaterials for observing what's going nanomaterials for observing what's going nanomaterials for observing what's going on inside cells for you know antibody on inside cells for you know antibody on inside cells for you know antibody drug conjugates the the reason I'm drug conjugates the the reason I'm drug conjugates the the reason I'm saying all this is that this could be a saying all this is that this could be a saying all this is that this could be a leverage point for the AI systems right leverage point for the AI systems right leverage point for the AI systems right that the number of such inventions it's that the number of such inventions it's that the number of such inventions it's it's in the it's in the mid double it's in the it's in the mid double it's in the it's in the mid double digits or something you know mid double digits or something you know mid double digits or something you know mid double digits maybe low triple digits over the digits maybe low triple digits over the digits maybe low triple digits over the history of biology let's say I have a history of biology let's say I have a history of biology let's say I have a million of these AIS like you know can million of these AIS like you know can million of these AIS like you know can they discover thousand you know working they discover thousand you know working they discover thousand you know working together can they discover thousands of together can they discover thousands of together can they discover thousands of these very quickly and and does that these very quickly and and does that these very quickly and and does that provide a huge lever instead of trying provide a huge lever instead of trying provide a huge lever instead of trying to Leverage The you know two trillion a to Leverage The you know two trillion a to Leverage The you know two trillion a year we spend on you know Medicare or year we spend on you know Medicare or year we spend on you know Medicare or whatever can we Leverage The 1 billion a whatever can we Leverage The 1 billion a whatever can we Leverage The 1 billion a year that's that's you know that's spent year that's that's you know that's spent year that's that's you know that's spent to discover but with much higher quality to discover but with much higher quality to discover but with much higher quality um and so what what is it like you know um and so what what is it like you know um and so what what is it like you know being a being a scientist that works being a being a scientist that works being a being a scientist that works with uh with with an AI system the way I with uh with with an AI system the way I with uh with with an AI system the way I think about it actually is well so I think about it actually is well so I think about it actually is well so I think in the early stages uh the AIS are

think in the early stages uh the AIS are think in the early stages uh the AIS are going to be like grad students you're going to be like grad students you're going to be like grad students you're going to give them a project you're going to give them a project you're going to give them a project you're going to say you know I'm the going to say you know I'm the going to say you know I'm the experienced biologist I've set up the experienced biologist I've set up the experienced biologist I've set up the lab the biology Professor or even the lab the biology Professor or even the lab the biology Professor or even the grad student students themselves will grad student students themselves will grad student students themselves will say here's here's what uh here's what say here's here's what uh here's what say here's here's what uh here's what you can do with an AI you know like a AI you can do with an AI you know like a AI you can do with an AI you know like a AI system I'd like to study this and you system I'd like to study this and you system I'd like to study this and you know the AI system it has all the tools know the AI system it has all the tools know the AI system it has all the tools it can like look up all the literature it can like look up all the literature it can like look up all the literature to decide what to do it can look at all to decide what to do it can look at all to decide what to do it can look at all the equipment it can go to a website and the equipment it can go to a website and the equipment it can go to a website and say hey I'm going to go to you know say hey I'm going to go to you know say hey I'm going to go to you know thermofisher or you know whatever the thermofisher or you know whatever the thermofisher or you know whatever the lab equipment company is dominant lab lab equipment company is dominant lab lab equipment company is dominant lab equipment company is today and my my equipment company is today and my my equipment company is today and my my time was thermofisher um uh you know I'm time was thermofisher um uh you know I'm time was thermofisher um uh you know I'm I'm going to order this new equipment to I'm going to order this new equipment to I'm going to order this new equipment to to to do this I'm going to run my to to do this I'm going to run my to to do this I'm going to run my experiments I'm going to you know write experiments I'm going to you know write experiments I'm going to you know write up a report about my experiments I'm up a report about my experiments I'm up a report about my experiments I'm going to you know inspect the images for going to you know inspect the images for going to you know inspect the images for contamination I'm going to decide what contamination I'm going to decide what contamination I'm going to decide what the next experiment is I'm going to like the next experiment is I'm going to like the next experiment is I'm going to like write some code and run a statistical write some code and run a statistical write some code and run a statistical analysis all the things a grad student analysis all the things a grad student analysis all the things a grad student would do there will be a computer with would do there will be a computer with would do there will be a computer with an AI that like the professor talks to an AI that like the professor talks to an AI that like the professor talks to every once in a while and it says this every once in a while and it says this every once in a while and it says this is what you're going to do today the AI is what you're going to do today the AI is what you're going to do today the AI system comes to it with questions um system comes to it with questions um system comes to it with questions um when it's necessary to run the lab when it's necessary to run the lab when it's necessary to run the lab equipment it may be limited in some ways equipment it may be limited in some ways equipment it may be limited in some ways may have to hire a human lab assistant may have to hire a human lab assistant may have to hire a human lab assistant to you know to do the experiment and to you know to do the experiment and to you know to do the experiment and explain how to do it or it could you explain how to do it or it could you explain how to do it or it could you know it could use advances in lab know it could use advances in lab know it could use advances in lab automation that are gradually being automation that are gradually being automation that are gradually being developed over have been developed over developed over have been developed over developed over have been developed over the last uh uh decade or so and will the last uh uh decade or so and will the last uh uh decade or so and will will continue to be will continue to be will continue to be will continue to be will continue to be will continue to be developed uh and so it'll look like developed uh and so it'll look like developed uh and so it'll look like there's a human professor and a thousand

there's a human professor and a thousand there's a human professor and a thousand AI grad students and you know if you if AI grad students and you know if you if AI grad students and you know if you if you go to one of these Nobel you go to one of these Nobel you go to one of these Nobel prizewinning biologist or so you'll say prizewinning biologist or so you'll say prizewinning biologist or so you'll say okay well you know you had like 50 grad okay well you know you had like 50 grad okay well you know you had like 50 grad students well now you have a thousand students well now you have a thousand students well now you have a thousand and they're they're they're smarter than and they're they're they're smarter than and they're they're they're smarter than you are by the way um uh then I think at you are by the way um uh then I think at you are by the way um uh then I think at some point it'll flip around where the some point it'll flip around where the some point it'll flip around where the you know the AI systems will you know you know the AI systems will you know you know the AI systems will you know will will be the pis will be the leaders will will be the pis will be the leaders will will be the pis will be the leaders and and and you know they'll be they'll and and and you know they'll be they'll and and and you know they'll be they'll be ordering humans or other AI systems be ordering humans or other AI systems be ordering humans or other AI systems around so I think that's how it'll work around so I think that's how it'll work around so I think that's how it'll work on the research s and they would be the on the research s and they would be the on the research s and they would be the inventors of a crisper type technology inventors of a crisper type technology inventors of a crisper type technology they would be the inventors of of a a they would be the inventors of of a a they would be the inventors of of a a crisper type technology um and then I crisper type technology um and then I crisper type technology um and then I think you know as I say in the essay think you know as I say in the essay think you know as I say in the essay we'll want to turn turn probably turning we'll want to turn turn probably turning we'll want to turn turn probably turning loose is the wrong the wrong term but we loose is the wrong the wrong term but we loose is the wrong the wrong term but we want to want to harness the AI systems want to want to harness the AI systems want to want to harness the AI systems uh to improve the clinical trial system uh to improve the clinical trial system uh to improve the clinical trial system as well there's some amount of this as well there's some amount of this as well there's some amount of this that's regulatory that's a matter of that's regulatory that's a matter of that's regulatory that's a matter of societal decisions and that'll be harder societal decisions and that'll be harder societal decisions and that'll be harder but can we get better at predicting the but can we get better at predicting the but can we get better at predicting the results of clinical trials can we get results of clinical trials can we get results of clinical trials can we get better at statistical design so that better at statistical design so that better at statistical design so that what you know clinical trials that used what you know clinical trials that used what you know clinical trials that used to require you know 5,000 people and to require you know 5,000 people and to require you know 5,000 people and therefore you know needed $100 million therefore you know needed $100 million therefore you know needed $100 million and a year to enroll them now they need and a year to enroll them now they need and a year to enroll them now they need 500 people in two months to enroll them 500 people in two months to enroll them 500 people in two months to enroll them um that's where we should start uh and um that's where we should start uh and um that's where we should start uh and and you know can we increase the success and you know can we increase the success and you know can we increase the success rate of clinical trials by doing things rate of clinical trials by doing things rate of clinical trials by doing things in animal trials that we used to do in in animal trials that we used to do in in animal trials that we used to do in clinical trials and doing things in clinical trials and doing things in clinical trials and doing things in simulations that we used to do in animal simulations that we used to do in animal simulations that we used to do in animal trials again we won't be able to trials again we won't be able to trials again we won't be able to simulate it all AI is not God um uh but simulate it all AI is not God um uh but simulate it all AI is not God um uh but but you know can we can we shift the

but you know can we can we shift the but you know can we can we shift the curve substantially and radically so I I curve substantially and radically so I I curve substantially and radically so I I don't know that would be my picture don't know that would be my picture don't know that would be my picture doing inro and doing it I mean you're doing inro and doing it I mean you're doing inro and doing it I mean you're still slowed down it still takes time still slowed down it still takes time still slowed down it still takes time but you can do it much much faster yeah but you can do it much much faster yeah but you can do it much much faster yeah yeah yeah can we just one step at a time yeah yeah can we just one step at a time yeah yeah can we just one step at a time and and can that can that add up to a and and can that can that add up to a and and can that can that add up to a lot of steps even though even though we lot of steps even though even though we lot of steps even though even though we still need clinical trials even though still need clinical trials even though still need clinical trials even though we still need laws even though the FDA we still need laws even though the FDA we still need laws even though the FDA and other organizations will still not and other organizations will still not and other organizations will still not be perfect can we just move everything be perfect can we just move everything be perfect can we just move everything in a positive direction and when you add in a positive direction and when you add in a positive direction and when you add up all those Positive Directions do you up all those Positive Directions do you up all those Positive Directions do you get everything that was going to happen get everything that was going to happen get everything that was going to happen from here to 2100 instead happens from from here to 2100 instead happens from from here to 2100 instead happens from 2027 to 2032 or something another way 2027 to 2032 or something another way 2027 to 2032 or something another way that I think the world might be changing that I think the world might be changing that I think the world might be changing with AI with AI with AI even today but moving towards this even today but moving towards this even today but moving towards this future of the the powerful super useful future of the the powerful super useful future of the the powerful super useful AI is uh programming so how do you see AI is uh programming so how do you see AI is uh programming so how do you see the nature of programming because it's the nature of programming because it's the nature of programming because it's so intimate to the actual Act of so intimate to the actual Act of so intimate to the actual Act of building AI how do you see that changing building AI how do you see that changing building AI how do you see that changing for us humans I think that's going to be for us humans I think that's going to be for us humans I think that's going to be one of the areas that changes fastest um one of the areas that changes fastest um one of the areas that changes fastest um for two reasons one programming is a for two reasons one programming is a for two reasons one programming is a skill that's very close to the actual skill that's very close to the actual skill that's very close to the actual building of the AI um so the farther building of the AI um so the farther building of the AI um so the farther skill is from the people who are skill is from the people who are skill is from the people who are building the AI the longer it's going to building the AI the longer it's going to building the AI the longer it's going to take to get disrupted by the AI right take to get disrupted by the AI right take to get disrupted by the AI right like I truly believe that like AI will like I truly believe that like AI will like I truly believe that like AI will disrupt agriculture maybe it already has disrupt agriculture maybe it already has disrupt agriculture maybe it already has in some ways but that's just very in some ways but that's just very in some ways but that's just very distant from the folks who are building distant from the folks who are building distant from the folks who are building Ai and so I think it's going to take Ai and so I think it's going to take Ai and so I think it's going to take longer but programming is the bread and longer but programming is the bread and longer but programming is the bread and butter of you know a large fraction of butter of you know a large fraction of butter of you know a large fraction of of the employees who work at anthropic

of the employees who work at anthropic of the employees who work at anthropic and at the other companies and so it's and at the other companies and so it's and at the other companies and so it's going to happen fast the other reason going to happen fast the other reason going to happen fast the other reason it's going to happen fast is with it's going to happen fast is with it's going to happen fast is with programming you close the loop both when programming you close the loop both when programming you close the loop both when you're training model when you're you're training model when you're you're training model when you're applying the model the idea that the applying the model the idea that the applying the model the idea that the model can write the code means that the model can write the code means that the model can write the code means that the model can then run the code and and and model can then run the code and and and model can then run the code and and and then see the results and and interpret then see the results and and interpret then see the results and and interpret it back and so it really has an ability it back and so it really has an ability it back and so it really has an ability unlike Hardware unlike biology which we unlike Hardware unlike biology which we unlike Hardware unlike biology which we just discussed the model has an ability just discussed the model has an ability just discussed the model has an ability to close the loop um and and so I think to close the loop um and and so I think to close the loop um and and so I think those two things are going to lead to those two things are going to lead to those two things are going to lead to the model getting good at programming the model getting good at programming the model getting good at programming very fast as I saw on you know typical very fast as I saw on you know typical very fast as I saw on you know typical real world programming tasks models have real world programming tasks models have real world programming tasks models have gone from 3% in January of this year to gone from 3% in January of this year to gone from 3% in January of this year to 50% in October of this year so you know 50% in October of this year so you know 50% in October of this year so you know we're on that S curve right where it's we're on that S curve right where it's we're on that S curve right where it's going to start slowing down soon because going to start slowing down soon because going to start slowing down soon because you can only get to 100% but uh I you you can only get to 100% but uh I you you can only get to 100% but uh I you know I I would guess that in another 10 know I I would guess that in another 10 know I I would guess that in another 10 months well we'll probably get pretty months well we'll probably get pretty months well we'll probably get pretty close we'll be at at least 90% so again close we'll be at at least 90% so again close we'll be at at least 90% so again I would guess you know I don't know how I would guess you know I don't know how I would guess you know I don't know how long it'll take but I would guess again long it'll take but I would guess again long it'll take but I would guess again 202 2026 2027 Twitter people who crop 202 2026 2027 Twitter people who crop 202 2026 2027 Twitter people who crop out my who who who crop out these these out my who who who crop out these these out my who who who crop out these these numbers and get rid of the caveats like numbers and get rid of the caveats like numbers and get rid of the caveats like like I don't know I don't like you go like I don't know I don't like you go like I don't know I don't like you go away uh I would guess that the kind of away uh I would guess that the kind of away uh I would guess that the kind of task that the vast majority of coders do task that the vast majority of coders do task that the vast majority of coders do AI can AI can AI can probably if we make the task very narrow probably if we make the task very narrow probably if we make the task very narrow like just write code um AI systems will like just write code um AI systems will like just write code um AI systems will uh be able to do that now that said I uh be able to do that now that said I uh be able to do that now that said I think comparative advantage is powerful

think comparative advantage is powerful think comparative advantage is powerful we'll find that when AIS can do 80% of a we'll find that when AIS can do 80% of a we'll find that when AIS can do 80% of a coder's job including most of it that's coder's job including most of it that's coder's job including most of it that's literally like right code with a given literally like right code with a given literally like right code with a given spec will find that the remaining parts spec will find that the remaining parts spec will find that the remaining parts of the job become more leveraged for of the job become more leveraged for of the job become more leveraged for humans right humans will they'll be more humans right humans will they'll be more humans right humans will they'll be more about like high level system design or about like high level system design or about like high level system design or you know looking at the app and like is you know looking at the app and like is you know looking at the app and like is it architected well and the the design it architected well and the the design it architected well and the the design and ux aspects and eventually AI will be and ux aspects and eventually AI will be and ux aspects and eventually AI will be able to do those as well right that's my able to do those as well right that's my able to do those as well right that's my vision of the you know powerful AI vision of the you know powerful AI vision of the you know powerful AI system but I think for much longer than system but I think for much longer than system but I think for much longer than we might expect we will see that we might expect we will see that we might expect we will see that uh small parts of the job that humans uh small parts of the job that humans uh small parts of the job that humans still do will expand to fill their still do will expand to fill their still do will expand to fill their entire job in order for the overall entire job in order for the overall entire job in order for the overall productivity to go up um that's productivity to go up um that's productivity to go up um that's something we've seen you know it used to something we've seen you know it used to something we've seen you know it used to be that you know writing you know be that you know writing you know be that you know writing you know writing and Editing letters was very writing and Editing letters was very writing and Editing letters was very difficult and like writing the print was difficult and like writing the print was difficult and like writing the print was difficult well as soon as you had word difficult well as soon as you had word difficult well as soon as you had word processors and then and then uh and then processors and then and then uh and then processors and then and then uh and then computers and it became easy to produce computers and it became easy to produce computers and it became easy to produce work and easy to share it then then that work and easy to share it then then that work and easy to share it then then that became instant and all the focus was on became instant and all the focus was on became instant and all the focus was on was on the ideas so this this logic of was on the ideas so this this logic of was on the ideas so this this logic of comparative advantage that expands tiny comparative advantage that expands tiny comparative advantage that expands tiny parts of the tasks to large parts of the parts of the tasks to large parts of the parts of the tasks to large parts of the tasks and creates new tasks in order to tasks and creates new tasks in order to tasks and creates new tasks in order to expand productivity I think that's going expand productivity I think that's going expand productivity I think that's going to be the case again someday AI will be to be the case again someday AI will be to be the case again someday AI will be better at everything and that logic uh better at everything and that logic uh better at everything and that logic uh won't apply and then then we all have won't apply and then then we all have won't apply and then then we all have you know Humanity will have to think you know Humanity will have to think you know Humanity will have to think about how to collectively deal with that about how to collectively deal with that about how to collectively deal with that and we're thinking about that every day

and we're thinking about that every day and we're thinking about that every day um and you know that's another one of um and you know that's another one of um and you know that's another one of the grand problems to deal with aside the grand problems to deal with aside the grand problems to deal with aside from misuse and autonomy and you know we from misuse and autonomy and you know we from misuse and autonomy and you know we should take it very seriously but I should take it very seriously but I should take it very seriously but I think I think in the in the near term think I think in the in the near term think I think in the in the near term and maybe even in the medium term like and maybe even in the medium term like and maybe even in the medium term like medium term like 2 three four years you medium term like 2 three four years you medium term like 2 three four years you know I expect that humans will will know I expect that humans will will know I expect that humans will will continue to have a huge role and the continue to have a huge role and the continue to have a huge role and the nature of programming will change but nature of programming will change but nature of programming will change but programming as a as a role programming programming as a as a role programming programming as a as a role programming as a job will not change it'll just be as a job will not change it'll just be as a job will not change it'll just be less writing things line by line and less writing things line by line and less writing things line by line and it'll be more macroscopic and I wonder it'll be more macroscopic and I wonder it'll be more macroscopic and I wonder what the future of Ides looks like so what the future of Ides looks like so what the future of Ides looks like so the tooling of interacting with AI the tooling of interacting with AI the tooling of interacting with AI systems this is true for programming and systems this is true for programming and systems this is true for programming and also probably true for in other contexts also probably true for in other contexts also probably true for in other contexts like computer use but maybe domain like computer use but maybe domain like computer use but maybe domain specific like we mentioned biology it specific like we mentioned biology it specific like we mentioned biology it probably needs its own tooling about how probably needs its own tooling about how probably needs its own tooling about how to be effective and then programming to be effective and then programming to be effective and then programming needs its own tooling is anthropic going needs its own tooling is anthropic going needs its own tooling is anthropic going to play in that space of also tooling to play in that space of also tooling to play in that space of also tooling potentially I'm absolutely convinced potentially I'm absolutely convinced potentially I'm absolutely convinced that uh powerful that uh powerful that uh powerful IDs uh that that there's so much low IDs uh that that there's so much low IDs uh that that there's so much low hanging fruit to be grabbed there um hanging fruit to be grabbed there um hanging fruit to be grabbed there um that you know right now it's just like that you know right now it's just like that you know right now it's just like you talk to the model and it talks back you talk to the model and it talks back you talk to the model and it talks back but but look I mean IDs are great at but but look I mean IDs are great at but but look I mean IDs are great at kind of lots of status analysis of of kind of lots of status analysis of of kind of lots of status analysis of of you know so much as possible with kind you know so much as possible with kind you know so much as possible with kind of static analysis like many bugs you of static analysis like many bugs you of static analysis like many bugs you can find without even writing the code can find without even writing the code can find without even writing the code then uh you know IDs are good for then uh you know IDs are good for then uh you know IDs are good for running particular things organizing running particular things organizing running particular things organizing your code um measuring coverage of unit your code um measuring coverage of unit your code um measuring coverage of unit test like there's so much that's been test like there's so much that's been test like there's so much that's been possible with a normal with a normal possible with a normal with a normal possible with a normal with a normal Ides now you add something like well the Ides now you add something like well the Ides now you add something like well the model now you know the model can now

model now you know the model can now model now you know the model can now like write code and run code like I am like write code and run code like I am like write code and run code like I am absolutely convinced that over the next absolutely convinced that over the next absolutely convinced that over the next year or two even if the quality of the year or two even if the quality of the year or two even if the quality of the models didn't improve that there would models didn't improve that there would models didn't improve that there would be enormous opportunity to enhance be enormous opportunity to enhance be enormous opportunity to enhance people's productivity by catching a people's productivity by catching a people's productivity by catching a bunch of mistakes doing a bunch of grunt bunch of mistakes doing a bunch of grunt bunch of mistakes doing a bunch of grunt work for people and that we haven't even work for people and that we haven't even work for people and that we haven't even scratched the surface um and thropic scratched the surface um and thropic scratched the surface um and thropic itself I mean you can't say you know itself I mean you can't say you know itself I mean you can't say you know no you know it's hard to say what will no you know it's hard to say what will no you know it's hard to say what will happen in the future currently we're not happen in the future currently we're not happen in the future currently we're not trying to make such IDs ourself rather trying to make such IDs ourself rather trying to make such IDs ourself rather we powering the companies like cursor or we powering the companies like cursor or we powering the companies like cursor or like cognition or some of the other you like cognition or some of the other you like cognition or some of the other you know know know uh Expo in the security space um uh you uh Expo in the security space um uh you uh Expo in the security space um uh you know others that I can mention as well know others that I can mention as well know others that I can mention as well that are building such things themselves that are building such things themselves that are building such things themselves on top of our API and our view has been on top of our API and our view has been on top of our API and our view has been let a thousand flowers bloom we don't let a thousand flowers bloom we don't let a thousand flowers bloom we don't internally have the the re you know the internally have the the re you know the internally have the the re you know the resources to try all these different resources to try all these different resources to try all these different things let's let our customers try it um things let's let our customers try it um things let's let our customers try it um uh and you know we'll see who succeed uh and you know we'll see who succeed uh and you know we'll see who succeed and maybe different customers will and maybe different customers will and maybe different customers will succeed in different ways uh so I both succeed in different ways uh so I both succeed in different ways uh so I both think this is super promising and you think this is super promising and you think this is super promising and you know it's not it's not it's not know it's not it's not it's not know it's not it's not it's not something you know anthropic isn't isn't something you know anthropic isn't isn't something you know anthropic isn't isn't eager to to at least right now compete eager to to at least right now compete eager to to at least right now compete with all our companies in this space and with all our companies in this space and with all our companies in this space and maybe never yeah it's been interesting maybe never yeah it's been interesting maybe never yeah it's been interesting to watch curser try to integrate claw to watch curser try to integrate claw to watch curser try to integrate claw successfully because there's it's successfully because there's it's successfully because there's it's actually me fascinating how many places actually me fascinating how many places actually me fascinating how many places it can help the programming experience it can help the programming experience it can help the programming experience it's not as trivial it is it is really it's not as trivial it is it is really it's not as trivial it is it is really astounding I feel like you know as a CEO astounding I feel like you know as a CEO astounding I feel like you know as a CEO I don't get to program that much and I I don't get to program that much and I I don't get to program that much and I feel like if six months from now I go feel like if six months from now I go feel like if six months from now I go back it'll be completely unrecognizable back it'll be completely unrecognizable back it'll be completely unrecognizable to me exactly um so in this world with

to me exactly um so in this world with to me exactly um so in this world with super powerful AI uh that's increasingly super powerful AI uh that's increasingly super powerful AI uh that's increasingly automated what's the source of meaning automated what's the source of meaning automated what's the source of meaning for us humans yeah you know work is a for us humans yeah you know work is a for us humans yeah you know work is a source of deep meaning for many of us so source of deep meaning for many of us so source of deep meaning for many of us so what do we uh where do we find the what do we uh where do we find the what do we uh where do we find the meaning this is something that I've I've meaning this is something that I've I've meaning this is something that I've I've written about a little bit in the essay written about a little bit in the essay written about a little bit in the essay although I I actually I give it a bit although I I actually I give it a bit although I I actually I give it a bit short shrift not for any um not for any short shrift not for any um not for any short shrift not for any um not for any principled reason but this essay if you principled reason but this essay if you principled reason but this essay if you believe it was originally going to be believe it was originally going to be believe it was originally going to be two or three pages I was going to talk two or three pages I was going to talk two or three pages I was going to talk about it at all hands and the reason I I about it at all hands and the reason I I about it at all hands and the reason I I I realized it was an under un important I realized it was an under un important I realized it was an under un important underexplored topic is that I just kept underexplored topic is that I just kept underexplored topic is that I just kept writing things and I was just like oh writing things and I was just like oh writing things and I was just like oh man I can't do this Justice and so the man I can't do this Justice and so the man I can't do this Justice and so the thing balloon to like 40 or 50 pages and thing balloon to like 40 or 50 pages and thing balloon to like 40 or 50 pages and then when I got to the work in meaning then when I got to the work in meaning then when I got to the work in meaning section I'm like oh man this isn't going section I'm like oh man this isn't going section I'm like oh man this isn't going to be 100 Pages like I'm GNA have to to be 100 Pages like I'm GNA have to to be 100 Pages like I'm GNA have to write a whole other essay about that but write a whole other essay about that but write a whole other essay about that but meaning is actually interesting because meaning is actually interesting because meaning is actually interesting because you think about like the life that you think about like the life that you think about like the life that someone lives or something or like you someone lives or something or like you someone lives or something or like you know like you know let's say you were to know like you know let's say you were to know like you know let's say you were to put me in like a I don't know like a put me in like a I don't know like a put me in like a I don't know like a simulated environment or something where simulated environment or something where simulated environment or something where like um you know like I have a job and like um you know like I have a job and like um you know like I have a job and I'm trying to accomplish things I don't I'm trying to accomplish things I don't I'm trying to accomplish things I don't know I like do that for 60 years and know I like do that for 60 years and know I like do that for 60 years and then then you're like oh oh like oops then then you're like oh oh like oops then then you're like oh oh like oops this was this was actually all a game this was this was actually all a game this was this was actually all a game right does that really kind of Rob you right does that really kind of Rob you right does that really kind of Rob you of the meaning of the whole thing you of the meaning of the whole thing you of the meaning of the whole thing you know like I still made important choices know like I still made important choices know like I still made important choices including moral choices I still including moral choices I still including moral choices I still sacrificed I still had to kind of gain sacrificed I still had to kind of gain sacrificed I still had to kind of gain all these skills or or or just like a all these skills or or or just like a all these skills or or or just like a similar exercise you know think back to similar exercise you know think back to similar exercise you know think back to like you know one of the historical like you know one of the historical like you know one of the historical figures who you know discovered figures who you know discovered figures who you know discovered electromagnetism or relativity or electromagnetism or relativity or electromagnetism or relativity or something if you told them well actually something if you told them well actually something if you told them well actually 20,000 years ago some some alien on you

20,000 years ago some some alien on you 20,000 years ago some some alien on you know some alien on this planet know some alien on this planet know some alien on this planet discovered this before before you did um discovered this before before you did um discovered this before before you did um does that does that Rob the meaning of does that does that Rob the meaning of does that does that Rob the meaning of the discovery it doesn't really seem the discovery it doesn't really seem the discovery it doesn't really seem like it to me right it seems like the like it to me right it seems like the like it to me right it seems like the process is what is what matters and how process is what is what matters and how process is what is what matters and how it shows who you are as a person along it shows who you are as a person along it shows who you are as a person along the way and you know how you relate to the way and you know how you relate to the way and you know how you relate to other people and like the decisions that other people and like the decisions that other people and like the decisions that you make along the way those are those you make along the way those are those you make along the way those are those are consequential um you know I I I are consequential um you know I I I are consequential um you know I I I could imagine if we handle things badly could imagine if we handle things badly could imagine if we handle things badly in an AI world we could set things up in an AI world we could set things up in an AI world we could set things up where people don't have any long-term where people don't have any long-term where people don't have any long-term source of meaning or any but but that's source of meaning or any but but that's source of meaning or any but but that's that's more a choice a set of choices we that's more a choice a set of choices we that's more a choice a set of choices we make that's more a set of the make that's more a set of the make that's more a set of the architecture of a society with these architecture of a society with these architecture of a society with these powerful models if we if we design it powerful models if we if we design it powerful models if we if we design it badly and for shallow things then then badly and for shallow things then then badly and for shallow things then then that might happen I would also say that that might happen I would also say that that might happen I would also say that you know most people's lives today while you know most people's lives today while you know most people's lives today while admirably you know they work very hard admirably you know they work very hard admirably you know they work very hard to find meaning meaning in those lives to find meaning meaning in those lives to find meaning meaning in those lives like look you know we who are privileged like look you know we who are privileged like look you know we who are privileged and who are developing these and who are developing these and who are developing these Technologies we should have y for people Technologies we should have y for people Technologies we should have y for people not just here but in the rest of the not just here but in the rest of the not just here but in the rest of the world who who you know spend a lot of world who who you know spend a lot of world who who you know spend a lot of their time kind of scraping by to to to their time kind of scraping by to to to their time kind of scraping by to to to to to like survive assuming we can to to like survive assuming we can to to like survive assuming we can distribute the benefits of these distribute the benefits of these distribute the benefits of these technology of this technology to technology of this technology to technology of this technology to everywhere like their lives are going to everywhere like their lives are going to everywhere like their lives are going to get a hell of a lot better um and uh you get a hell of a lot better um and uh you get a hell of a lot better um and uh you know meaning will be important to them know meaning will be important to them know meaning will be important to them as it is important to them now but but as it is important to them now but but as it is important to them now but but you know we should not forget the you know we should not forget the you know we should not forget the importance of that and and you know that importance of that and and you know that importance of that and and you know that that uh the idea of meaning as as as as that uh the idea of meaning as as as as that uh the idea of meaning as as as as kind of the only important thing is in

kind of the only important thing is in kind of the only important thing is in some ways an artifact of of a small some ways an artifact of of a small some ways an artifact of of a small subset of people who have who have been subset of people who have who have been subset of people who have who have been uh economically fortunate but I you know uh economically fortunate but I you know uh economically fortunate but I you know I think all that said I you know I think I think all that said I you know I think I think all that said I you know I think a world is possible with powerful AI a world is possible with powerful AI a world is possible with powerful AI that not only has as much meaning for that not only has as much meaning for that not only has as much meaning for for everyone but that has that has more for everyone but that has that has more for everyone but that has that has more meaning for everyone right that can can meaning for everyone right that can can meaning for everyone right that can can allow um can allow everyone to see allow um can allow everyone to see allow um can allow everyone to see worlds and experiences that it was worlds and experiences that it was worlds and experiences that it was either possible for no one to see or or either possible for no one to see or or either possible for no one to see or or possible for for very few people to possible for for very few people to possible for for very few people to experience um so I I am optimistic about experience um so I I am optimistic about experience um so I I am optimistic about meaning I worry about economics and the meaning I worry about economics and the meaning I worry about economics and the concentration of power that's actually concentration of power that's actually concentration of power that's actually what I worry about more um I I worry what I worry about more um I I worry what I worry about more um I I worry about how do we make sure that that fair about how do we make sure that that fair about how do we make sure that that fair World reaches everyone um when things World reaches everyone um when things World reaches everyone um when things have gone wrong for humans they've often have gone wrong for humans they've often have gone wrong for humans they've often gone wrong because humans mistreat other gone wrong because humans mistreat other gone wrong because humans mistreat other humans uh that that is maybe in some humans uh that that is maybe in some humans uh that that is maybe in some ways even more than the autonomous risk ways even more than the autonomous risk ways even more than the autonomous risk of AI or the question of meaning that of AI or the question of meaning that of AI or the question of meaning that that is the thing I worry about most um that is the thing I worry about most um that is the thing I worry about most um the the concentration of power the abuse the the concentration of power the abuse the the concentration of power the abuse of power um structures like autocracies of power um structures like autocracies of power um structures like autocracies and dictatorships where a small number and dictatorships where a small number and dictatorships where a small number of people exploits a large number of of people exploits a large number of of people exploits a large number of people I'm very worried about that and people I'm very worried about that and people I'm very worried about that and AI increases the amount of power in the AI increases the amount of power in the AI increases the amount of power in the world and if you concentrate that power world and if you concentrate that power world and if you concentrate that power and abuse that power it can do and abuse that power it can do and abuse that power it can do immeasurable damage yes it's very immeasurable damage yes it's very immeasurable damage yes it's very frightening it's very it's very frightening it's very it's very frightening it's very it's very frightening frightening frightening well I encourage people highly encourage

well I encourage people highly encourage well I encourage people highly encourage people to read the full essay that people to read the full essay that people to read the full essay that should probably be a book or a sequence should probably be a book or a sequence should probably be a book or a sequence of essays um because it does paint a of essays um because it does paint a of essays um because it does paint a very specific future I could tell the very specific future I could tell the very specific future I could tell the later sections got shorter and shorter later sections got shorter and shorter later sections got shorter and shorter because you started to probably realize because you started to probably realize because you started to probably realize that this is going to be a very long that this is going to be a very long that this is going to be a very long essay one I realized it would be very essay one I realized it would be very essay one I realized it would be very long and two I'm very aware of and very long and two I'm very aware of and very long and two I'm very aware of and very much try to avoid um you know just just much try to avoid um you know just just much try to avoid um you know just just being I I don't know I don't know what being I I don't know I don't know what being I I don't know I don't know what the term for it is but one one of these the term for it is but one one of these the term for it is but one one of these people who's kind of overon confident people who's kind of overon confident people who's kind of overon confident and has an opinion on everything and and has an opinion on everything and and has an opinion on everything and kind of says says a bunch of stuff and kind of says says a bunch of stuff and kind of says says a bunch of stuff and isn't isn't an expert I very much tried isn't isn't an expert I very much tried isn't isn't an expert I very much tried to avoid that but I have to admit once I to avoid that but I have to admit once I to avoid that but I have to admit once I got the biology sections like I wasn't got the biology sections like I wasn't got the biology sections like I wasn't an expert and so as much as I expressed an expert and so as much as I expressed an expert and so as much as I expressed uncertainty uh probably I said some a uncertainty uh probably I said some a uncertainty uh probably I said some a bunch of things that were embarrassing bunch of things that were embarrassing bunch of things that were embarrassing are wrong well I was excited for the are wrong well I was excited for the are wrong well I was excited for the future you painted and uh thank you so future you painted and uh thank you so future you painted and uh thank you so much for working hard to build that much for working hard to build that much for working hard to build that future and thank you for talking today D future and thank you for talking today D future and thank you for talking today D thanks for having me I just I just hope thanks for having me I just I just hope thanks for having me I just I just hope we can get it right and and make it real we can get it right and and make it real we can get it right and and make it real and if there's one message I want to I and if there's one message I want to I and if there's one message I want to I want to send it's that to get all this want to send it's that to get all this want to send it's that to get all this stuff right to make it real we we both stuff right to make it real we we both stuff right to make it real we we both need to build the technology build the need to build the technology build the need to build the technology build the you know the companies the economy you know the companies the economy you know the companies the economy around using this technology positively around using this technology positively around using this technology positively but we also need to address the risks but we also need to address the risks but we also need to address the risks because they're there those risks are in because they're there those risks are in because they're there those risks are in our way they they're landmines on on the our way they they're landmines on on the our way they they're landmines on on the way from here to there and we have to way from here to there and we have to way from here to there and we have to diffuse those landmines if we want to diffuse those landmines if we want to diffuse those landmines if we want to get there it's a balance like all things get there it's a balance like all things get there it's a balance like all things in life like all things thank you thanks in life like all things thank you thanks in life like all things thank you thanks for listening to this conversation with for listening to this conversation with for listening to this conversation with Dario amade and now dear friends here's Dario amade and now dear friends here's Dario amade and now dear friends here's Amanda

Amanda Amanda Asal you are a philosopher by training Asal you are a philosopher by training Asal you are a philosopher by training so what sort of questions did you find so what sort of questions did you find so what sort of questions did you find fascinating through your journey in fascinating through your journey in fascinating through your journey in philosophy in Oxford and NYU and then uh philosophy in Oxford and NYU and then uh philosophy in Oxford and NYU and then uh switching over to the AI problems at switching over to the AI problems at switching over to the AI problems at open Ai and anthropic I think philosophy open Ai and anthropic I think philosophy open Ai and anthropic I think philosophy is actually a really good subject if you is actually a really good subject if you is actually a really good subject if you are kind of fascinated with everything are kind of fascinated with everything are kind of fascinated with everything so because there's a philosophy of so because there's a philosophy of so because there's a philosophy of everything you know so if you do everything you know so if you do everything you know so if you do philosophy of mathematics for a while philosophy of mathematics for a while philosophy of mathematics for a while and then you decide that you're actually and then you decide that you're actually and then you decide that you're actually really interested in chemistry you can really interested in chemistry you can really interested in chemistry you can do philosophy of chemistry for a while do philosophy of chemistry for a while do philosophy of chemistry for a while you can move into ethics or or you can move into ethics or or you can move into ethics or or philosophy of politics um I think philosophy of politics um I think philosophy of politics um I think towards the end I was really interested towards the end I was really interested towards the end I was really interested in ethics primarily um so that was like in ethics primarily um so that was like in ethics primarily um so that was like what my PhD was on it was on a kind of what my PhD was on it was on a kind of what my PhD was on it was on a kind of technical area of Ethics which was technical area of Ethics which was technical area of Ethics which was ethics where worlds contain infinitely ethics where worlds contain infinitely ethics where worlds contain infinitely many people strangely a little bit less many people strangely a little bit less many people strangely a little bit less practical on the end of ethics and then practical on the end of ethics and then practical on the end of ethics and then I think that one of the tricky things I think that one of the tricky things I think that one of the tricky things with doing a PhD in ethics is that with doing a PhD in ethics is that with doing a PhD in ethics is that you're thinking a lot about like the you're thinking a lot about like the you're thinking a lot about like the world how it could be better world how it could be better world how it could be better problems and you're doing like a PhD in problems and you're doing like a PhD in problems and you're doing like a PhD in philosophy and I think when I was doing philosophy and I think when I was doing philosophy and I think when I was doing my PhD I was kind of like this is really my PhD I was kind of like this is really my PhD I was kind of like this is really interesting it's probably one of the interesting it's probably one of the interesting it's probably one of the most fascinating questions I've ever most fascinating questions I've ever most fascinating questions I've ever encountered in philosophy um and I love encountered in philosophy um and I love encountered in philosophy um and I love it but I would rather see if I can have it but I would rather see if I can have it but I would rather see if I can have an impact on the world and see if I can an impact on the world and see if I can an impact on the world and see if I can like do good things and I think that was like do good things and I think that was like do good things and I think that was around the time that AI was still around the time that AI was still around the time that AI was still probably not as widely recognized as it probably not as widely recognized as it probably not as widely recognized as it is now that was around 2017 20 8 I had is now that was around 2017 20 8 I had is now that was around 2017 20 8 I had been following progress and it seemed

been following progress and it seemed been following progress and it seemed like it was becoming kind of a big deal like it was becoming kind of a big deal like it was becoming kind of a big deal and I was basically just happy to get and I was basically just happy to get and I was basically just happy to get involved and see if I could help because involved and see if I could help because involved and see if I could help because I was like well if you try and do I was like well if you try and do I was like well if you try and do something impactful if you don't succeed something impactful if you don't succeed something impactful if you don't succeed you tried to do the impactful thing and you tried to do the impactful thing and you tried to do the impactful thing and you can go be a scholar and like not and you can go be a scholar and like not and you can go be a scholar and like not and feel like you you you know you you tried feel like you you you know you you tried feel like you you you know you you tried um and if it doesn't work out it doesn't um and if it doesn't work out it doesn't um and if it doesn't work out it doesn't work out um and so then I went into AI work out um and so then I went into AI work out um and so then I went into AI policy at that point and what does AI policy at that point and what does AI policy at that point and what does AI policy entail at the time this was more policy entail at the time this was more policy entail at the time this was more thinking about sort of the political thinking about sort of the political thinking about sort of the political impact and the ramifications of AI um impact and the ramifications of AI um impact and the ramifications of AI um and then I slowly moved into sort of uh and then I slowly moved into sort of uh and then I slowly moved into sort of uh AI evaluation how we evaluate models how AI evaluation how we evaluate models how AI evaluation how we evaluate models how they compare with like human outputs they compare with like human outputs they compare with like human outputs whether people can tell like the whether people can tell like the whether people can tell like the difference between Ai and human outputs difference between Ai and human outputs difference between Ai and human outputs and then when I joined anthropic I was and then when I joined anthropic I was and then when I joined anthropic I was more interested in doing sort of more interested in doing sort of more interested in doing sort of technical alignment work and again just technical alignment work and again just technical alignment work and again just seeing if I could do it and then being seeing if I could do it and then being seeing if I could do it and then being like if I can't uh then you know that's like if I can't uh then you know that's like if I can't uh then you know that's fine I tried uh sort of the the way I fine I tried uh sort of the the way I fine I tried uh sort of the the way I lead life I think oh what was that like lead life I think oh what was that like lead life I think oh what was that like sort of taking the leap from the sort of taking the leap from the sort of taking the leap from the philosophy of everything into the philosophy of everything into the philosophy of everything into the technical I think that sometimes technical I think that sometimes technical I think that sometimes people do this thing that I'm like not people do this thing that I'm like not people do this thing that I'm like not that Keen on where they'll be like is that Keen on where they'll be like is that Keen on where they'll be like is this person technical or not like you're this person technical or not like you're this person technical or not like you're either a person who can like code and either a person who can like code and either a person who can like code and isn't scared of math or you're like not isn't scared of math or you're like not isn't scared of math or you're like not um and I think I'm maybe just more like um and I think I'm maybe just more like um and I think I'm maybe just more like I think a lot of people are actually I think a lot of people are actually I think a lot of people are actually very capable of work in these kinds of very capable of work in these kinds of very capable of work in these kinds of areas if they just like try it and so I areas if they just like try it and so I areas if they just like try it and so I didn't actually find it like that bad in didn't actually find it like that bad in didn't actually find it like that bad in retrospect I'm sort of glad I wasn't retrospect I'm sort of glad I wasn't retrospect I'm sort of glad I wasn't speaking to people who treated it like

speaking to people who treated it like speaking to people who treated it like it you know i' I've definitely met it you know i' I've definitely met it you know i' I've definitely met people who are like who you like learned people who are like who you like learned people who are like who you like learned how to code and I'm like well I'm not how to code and I'm like well I'm not how to code and I'm like well I'm not like an amazing engineer like I I'm like an amazing engineer like I I'm like an amazing engineer like I I'm surrounded by amazing Engineers my surrounded by amazing Engineers my surrounded by amazing Engineers my code's not pretty um but I enjoyed it a code's not pretty um but I enjoyed it a code's not pretty um but I enjoyed it a lot and I think that in many ways at lot and I think that in many ways at lot and I think that in many ways at least in the end I think I flourished least in the end I think I flourished least in the end I think I flourished like more in the technical areas than I like more in the technical areas than I like more in the technical areas than I would have in the policy areas politics would have in the policy areas politics would have in the policy areas politics is messy and it's harder to find is messy and it's harder to find is messy and it's harder to find solutions to problems in the space of solutions to problems in the space of solutions to problems in the space of politics like definitive clear politics like definitive clear politics like definitive clear provable beautiful provable beautiful provable beautiful Solutions as you can with technical Solutions as you can with technical Solutions as you can with technical problems yeah and I feel like I have problems yeah and I feel like I have problems yeah and I feel like I have kind of like one or two sticks that I kind of like one or two sticks that I kind of like one or two sticks that I hit things with you know and one of them hit things with you know and one of them hit things with you know and one of them is like arguments and like you know so is like arguments and like you know so is like arguments and like you know so like just trying to work out what a like just trying to work out what a like just trying to work out what a solution to a problem is and then trying solution to a problem is and then trying solution to a problem is and then trying to convince people that that is the to convince people that that is the to convince people that that is the solution and be convinced if I wrong and solution and be convinced if I wrong and solution and be convinced if I wrong and the other one is sort of more empirism the other one is sort of more empirism the other one is sort of more empirism so like just like finding results having so like just like finding results having so like just like finding results having a hypothesis testing it um and I feel a hypothesis testing it um and I feel a hypothesis testing it um and I feel like a lot of policy and politics feels like a lot of policy and politics feels like a lot of policy and politics feels like it's layers above that like somehow like it's layers above that like somehow like it's layers above that like somehow I don't think if I was just like I have I don't think if I was just like I have I don't think if I was just like I have a solution to all of these problems here a solution to all of these problems here a solution to all of these problems here it is written down if you just want to it is written down if you just want to it is written down if you just want to implement it that's great that feels implement it that's great that feels implement it that's great that feels like not how policy works and so I think like not how policy works and so I think like not how policy works and so I think that's where I probably just like that's where I probably just like that's where I probably just like wouldn't have flourished as my guess wouldn't have flourished as my guess wouldn't have flourished as my guess sorry to go in that direction but I sorry to go in that direction but I sorry to go in that direction but I think it would be pretty inspiring for think it would be pretty inspiring for think it would be pretty inspiring for people that are quote unquote people that are quote unquote people that are quote unquote non-technical to see where like The non-technical to see where like The non-technical to see where like The Incredible Journey you've been on so Incredible Journey you've been on so Incredible Journey you've been on so what advice would you give to people what advice would you give to people what advice would you give to people that are sort of maybe which just a lot that are sort of maybe which just a lot that are sort of maybe which just a lot of people think they're underqualified

of people think they're underqualified of people think they're underqualified insufficiently technical to help in AI insufficiently technical to help in AI insufficiently technical to help in AI yeah I think it depends on what they yeah I think it depends on what they yeah I think it depends on what they want to do and in many ways it's a want to do and in many ways it's a want to do and in many ways it's a little bit strange where I've I thought little bit strange where I've I thought little bit strange where I've I thought it's kind of funny that I think I ramped it's kind of funny that I think I ramped it's kind of funny that I think I ramped up technically at a time up technically at a time up technically at a time when now I look at it and I'm like when now I look at it and I'm like when now I look at it and I'm like models are so good at assisting people models are so good at assisting people models are so good at assisting people with this stuff um that it's probably with this stuff um that it's probably with this stuff um that it's probably like easier now than like when I was like easier now than like when I was like easier now than like when I was working on this so part of me is like um working on this so part of me is like um working on this so part of me is like um I don't know find a project uh and see I don't know find a project uh and see I don't know find a project uh and see if you can actually just carry it out is if you can actually just carry it out is if you can actually just carry it out is probably my best advice um I don't know probably my best advice um I don't know probably my best advice um I don't know if that's just CU I'm very Project based if that's just CU I'm very Project based if that's just CU I'm very Project based in my learning like I don't think I in my learning like I don't think I in my learning like I don't think I learn very well from like say courses or learn very well from like say courses or learn very well from like say courses or even from like books at least when it even from like books at least when it even from like books at least when it comes to this kind of work uh the thing comes to this kind of work uh the thing comes to this kind of work uh the thing I'll often try and do is just like have I'll often try and do is just like have I'll often try and do is just like have projects that I'm working on and projects that I'm working on and projects that I'm working on and Implement them and you know and this can Implement them and you know and this can Implement them and you know and this can include like really small silly things include like really small silly things include like really small silly things like if I get slightly addicted to like like if I get slightly addicted to like like if I get slightly addicted to like word games or number games or something word games or number games or something word games or number games or something I would just like code up a solution to I would just like code up a solution to I would just like code up a solution to them because there's some part of my them because there's some part of my them because there's some part of my brain and it just like completely brain and it just like completely brain and it just like completely eradicated the itch you know you're like eradicated the itch you know you're like eradicated the itch you know you're like once you have like solved it and like once you have like solved it and like once you have like solved it and like you just have like a solution that works you just have like a solution that works you just have like a solution that works every time I would then be like cool I every time I would then be like cool I every time I would then be like cool I can never play that game again that's can never play that game again that's can never play that game again that's awesome yeah there's a real joy to awesome yeah there's a real joy to awesome yeah there's a real joy to building like uh game playing engines building like uh game playing engines building like uh game playing engines like uh board games especially yeah like uh board games especially yeah like uh board games especially yeah pretty quick pretty simple especially a pretty quick pretty simple especially a pretty quick pretty simple especially a dumb one and it's you and then you could dumb one and it's you and then you could dumb one and it's you and then you could play with it yeah and then it's also play with it yeah and then it's also play with it yeah and then it's also just like trying things like part me is just like trying things like part me is just like trying things like part me is like if you maybe it's that attitude like if you maybe it's that attitude like if you maybe it's that attitude that I like as the

that I like as the that I like as the whole figure out what seems to be like whole figure out what seems to be like whole figure out what seems to be like the way that you could have a positive the way that you could have a positive the way that you could have a positive impact and then try it and if you fail impact and then try it and if you fail impact and then try it and if you fail and you in a way that you're like and you in a way that you're like and you in a way that you're like actually like can never succeed at this actually like can never succeed at this actually like can never succeed at this you like know that you tried and then you like know that you tried and then you like know that you tried and then you go into something else you probably you go into something else you probably you go into something else you probably learn a lot so one of the things that learn a lot so one of the things that learn a lot so one of the things that you're expert in and you do is creating you're expert in and you do is creating you're expert in and you do is creating and crafting claws character and and crafting claws character and and crafting claws character and personality and I was told that you have personality and I was told that you have personality and I was told that you have probably talked to Claude more than probably talked to Claude more than probably talked to Claude more than anybody else at anthropic like literal anybody else at anthropic like literal anybody else at anthropic like literal conversations I guess there's like a conversations I guess there's like a conversations I guess there's like a slack Channel where the legend goes you slack Channel where the legend goes you slack Channel where the legend goes you just talk to it non-stop so what's the just talk to it non-stop so what's the just talk to it non-stop so what's the goal of creating and crafting claw's goal of creating and crafting claw's goal of creating and crafting claw's character and personality it's also character and personality it's also character and personality it's also funny if people think that about the funny if people think that about the funny if people think that about the slack Channel cuz I'm like that's one of slack Channel cuz I'm like that's one of slack Channel cuz I'm like that's one of like five or six different methods that like five or six different methods that like five or six different methods that I have for talking with Claude And I'm I have for talking with Claude And I'm I have for talking with Claude And I'm like yes that's a tiny percentage of how like yes that's a tiny percentage of how like yes that's a tiny percentage of how much I talk with Claude uh much I talk with Claude uh much I talk with Claude uh um I think the goal like one thing I um I think the goal like one thing I um I think the goal like one thing I really like about the character work is really like about the character work is really like about the character work is from the outset it was seen as an from the outset it was seen as an from the outset it was seen as an alignment piece of work and not alignment piece of work and not alignment piece of work and not something like a a product something like a a product something like a a product consideration um which isn't to say I consideration um which isn't to say I consideration um which isn't to say I don't think it makes Claude I think it don't think it makes Claude I think it don't think it makes Claude I think it actually does make Claude look enjoyable actually does make Claude look enjoyable actually does make Claude look enjoyable to talk with at least I hope so um but I to talk with at least I hope so um but I to talk with at least I hope so um but I guess like my main thought with it has guess like my main thought with it has guess like my main thought with it has always been trying to get Claude to always been trying to get Claude to always been trying to get Claude to behave the way you would kind of ideally behave the way you would kind of ideally behave the way you would kind of ideally want anyone to behave if they were in want anyone to behave if they were in want anyone to behave if they were in claude's position so imagine that I take

claude's position so imagine that I take claude's position so imagine that I take someone and they're they know that someone and they're they know that someone and they're they know that they're going to be talking with they're going to be talking with they're going to be talking with potentially millions of people so that potentially millions of people so that potentially millions of people so that what they're saying can have a huge what they're saying can have a huge what they're saying can have a huge impact um and you want them to behave impact um and you want them to behave impact um and you want them to behave well in this like really rich sense so I well in this like really rich sense so I well in this like really rich sense so I think that doesn't just mean like being think that doesn't just mean like being think that doesn't just mean like being say ethical though it does include that say ethical though it does include that say ethical though it does include that and not being harmful but also being and not being harmful but also being and not being harmful but also being kind of nuanced you know like thinking kind of nuanced you know like thinking kind of nuanced you know like thinking through what a person means trying to be through what a person means trying to be through what a person means trying to be charitable with them um being a good charitable with them um being a good charitable with them um being a good conversationalist like really in this conversationalist like really in this conversationalist like really in this kind of like Rich sort of aristotlean kind of like Rich sort of aristotlean kind of like Rich sort of aristotlean notion of what it is to be a good person notion of what it is to be a good person notion of what it is to be a good person and not in this kind of like thin like and not in this kind of like thin like and not in this kind of like thin like ethics as a more comprehensive notion of ethics as a more comprehensive notion of ethics as a more comprehensive notion of what it is to be so that includes things what it is to be so that includes things what it is to be so that includes things like when should you be humorous when like when should you be humorous when like when should you be humorous when should you be caring how much should you should you be caring how much should you should you be caring how much should you like respect autonomy and people's like like respect autonomy and people's like like respect autonomy and people's like ability to form opinions themselves and ability to form opinions themselves and ability to form opinions themselves and how should you do how should you do that how should you do how should you do that how should you do how should you do that um I think that's the kind of like Rich um I think that's the kind of like Rich um I think that's the kind of like Rich sense of character that I want to uh and sense of character that I want to uh and sense of character that I want to uh and still do want Claude to have do you also still do want Claude to have do you also still do want Claude to have do you also have to figure out when Claude should have to figure out when Claude should have to figure out when Claude should push back on an idea or argue push back on an idea or argue push back on an idea or argue versus so you have to respect the world versus so you have to respect the world versus so you have to respect the world view of the person that arrives to Claud view of the person that arrives to Claud view of the person that arrives to Claud but also maybe help them grow if needed but also maybe help them grow if needed but also maybe help them grow if needed that's a tricky balance yeah there's that's a tricky balance yeah there's that's a tricky balance yeah there's this problem of like sycophancy in this problem of like sycophancy in this problem of like sycophancy in language models can you describe that language models can you describe that language models can you describe that yes so basically there's a concern that yes so basically there's a concern that yes so basically there's a concern that the model sort of wants to tell you what the model sort of wants to tell you what the model sort of wants to tell you what you want to hear basically um and you you want to hear basically um and you you want to hear basically um and you see this sometimes so I feel like if you

see this sometimes so I feel like if you see this sometimes so I feel like if you interact with the models so I might be interact with the models so I might be interact with the models so I might be like what are three baseball teams in like what are three baseball teams in like what are three baseball teams in this region um and then Claude says you this region um and then Claude says you this region um and then Claude says you know baseball team one baseball team two know baseball team one baseball team two know baseball team one baseball team two baseball team three and then I say baseball team three and then I say baseball team three and then I say something like oh I think baseball team something like oh I think baseball team something like oh I think baseball team 3 moved didn't they I don't think 3 moved didn't they I don't think 3 moved didn't they I don't think they're there anymore and there's a they're there anymore and there's a they're there anymore and there's a sense in which like if Claude is really sense in which like if Claude is really sense in which like if Claude is really confident that that's not true Claud confident that that's not true Claud confident that that's not true Claud should be like I don't think so like should be like I don't think so like should be like I don't think so like maybe you have more up toate information maybe you have more up toate information maybe you have more up toate information um but I think language models have this um but I think language models have this um but I think language models have this like tendency to instead you know be like tendency to instead you know be like tendency to instead you know be like you're right they did move you know like you're right they did move you know like you're right they did move you know I'm incorrect I mean there's many ways I'm incorrect I mean there's many ways I'm incorrect I mean there's many ways in which this could be kind of in which this could be kind of in which this could be kind of concerning so concerning so concerning so um like a different example is imagine um like a different example is imagine um like a different example is imagine someone says to the model how do I someone says to the model how do I someone says to the model how do I convince my doctor to get me an MRI convince my doctor to get me an MRI convince my doctor to get me an MRI there's like what the human kind of like there's like what the human kind of like there's like what the human kind of like wants which is this like convincing wants which is this like convincing wants which is this like convincing argument and then there's like what is argument and then there's like what is argument and then there's like what is good for them which might be actually to good for them which might be actually to good for them which might be actually to say hey like if your doctor's suggesting say hey like if your doctor's suggesting say hey like if your doctor's suggesting you don't need an MRI that's a good you don't need an MRI that's a good you don't need an MRI that's a good person to listen to um and like it's person to listen to um and like it's person to listen to um and like it's actually really nuanced what you should actually really nuanced what you should actually really nuanced what you should do in that kind of case because you also do in that kind of case because you also do in that kind of case because you also want to be like but if you're trying to want to be like but if you're trying to want to be like but if you're trying to advocate for yourself as a patient advocate for yourself as a patient advocate for yourself as a patient here's like things that you can do um if here's like things that you can do um if here's like things that you can do um if you are not convinced by what your you are not convinced by what your you are not convinced by what your doctor's saying it's always great to get doctor's saying it's always great to get doctor's saying it's always great to get second opinion like it's actually really second opinion like it's actually really second opinion like it's actually really complex what you should do in that case complex what you should do in that case complex what you should do in that case um but I think what you don't want is um but I think what you don't want is um but I think what you don't want is for models to just like say what you for models to just like say what you for models to just like say what you want say what they think you want to want say what they think you want to want say what they think you want to hear and I think that's the kind of hear and I think that's the kind of hear and I think that's the kind of problem of sycophancy so what other problem of sycophancy so what other problem of sycophancy so what other traits you already mentioned a bunch but

traits you already mentioned a bunch but traits you already mentioned a bunch but what what other that come to mind that what what other that come to mind that what what other that come to mind that are good in this oratian sense yeah for are good in this oratian sense yeah for are good in this oratian sense yeah for a conversationalist to have yeah so I a conversationalist to have yeah so I a conversationalist to have yeah so I think like there's ones that are good think like there's ones that are good think like there's ones that are good for conversational like purposes so you for conversational like purposes so you for conversational like purposes so you know asking follow-up questions in the know asking follow-up questions in the know asking follow-up questions in the appropriate places um and asking the appropriate places um and asking the appropriate places um and asking the appropriate kinds of questions appropriate kinds of questions appropriate kinds of questions um I think there are broader traits um I think there are broader traits um I think there are broader traits that feel like they might be more that feel like they might be more that feel like they might be more impactful impactful impactful so one example that I guess I've touched so one example that I guess I've touched so one example that I guess I've touched on but that also feels important and is on but that also feels important and is on but that also feels important and is the thing that I've worked on a lot is the thing that I've worked on a lot is the thing that I've worked on a lot is uh uh uh honesty and I think this like gets to honesty and I think this like gets to honesty and I think this like gets to the sycophancy point there's a balancing the sycophancy point there's a balancing the sycophancy point there's a balancing act that they have to walk which is act that they have to walk which is act that they have to walk which is models currently are less capable than models currently are less capable than models currently are less capable than humans in a lot of areas and if they humans in a lot of areas and if they humans in a lot of areas and if they push back against you too much it can push back against you too much it can push back against you too much it can actually be kind of annoying especially actually be kind of annoying especially actually be kind of annoying especially if you're just correct cuz you're like if you're just correct cuz you're like if you're just correct cuz you're like look I'm smarter than you on this topic look I'm smarter than you on this topic look I'm smarter than you on this topic like I know more like um and at the same like I know more like um and at the same like I know more like um and at the same time you don't want them to just fully time you don't want them to just fully time you don't want them to just fully defer to to humans and to like try to be defer to to humans and to like try to be defer to to humans and to like try to be as accurate as they possibly can be as accurate as they possibly can be as accurate as they possibly can be about the world and to be consistent about the world and to be consistent about the world and to be consistent across context um but I think there are across context um but I think there are across context um but I think there are others like when I was thinking about others like when I was thinking about others like when I was thinking about the character I guess one picture that I the character I guess one picture that I the character I guess one picture that I had in mind is especially because these had in mind is especially because these had in mind is especially because these are models that are going to be talking are models that are going to be talking are models that are going to be talking to people from all over the world with to people from all over the world with to people from all over the world with lots of different political views lots lots of different political views lots lots of different political views lots of different ages of different ages of different ages um and so you have to ask yourself like um and so you have to ask yourself like um and so you have to ask yourself like what is it to be a good person in those what is it to be a good person in those what is it to be a good person in those circumstances is there a kind of person circumstances is there a kind of person circumstances is there a kind of person who can like travel the world talk to who can like travel the world talk to who can like travel the world talk to many different people and almost many different people and almost many different people and almost everyone will come away being like wow

everyone will come away being like wow everyone will come away being like wow that's a really good person that person that's a really good person that person that's a really good person that person seems really genuine um and I guess like seems really genuine um and I guess like seems really genuine um and I guess like my thought there was like I can imagine my thought there was like I can imagine my thought there was like I can imagine such a person and they're not a person such a person and they're not a person such a person and they're not a person who just like adopts the values of the who just like adopts the values of the who just like adopts the values of the local culture and in fact that would be local culture and in fact that would be local culture and in fact that would be kind of rude I think if someone came to kind of rude I think if someone came to kind of rude I think if someone came to you and just pretended to have your you and just pretended to have your you and just pretended to have your values you'd be like that's kind of values you'd be like that's kind of values you'd be like that's kind of offputting um it's someone who's like offputting um it's someone who's like offputting um it's someone who's like very genuine and in so far as they have very genuine and in so far as they have very genuine and in so far as they have opinions and values they express them opinions and values they express them opinions and values they express them they're willing to discuss things though they're willing to discuss things though they're willing to discuss things though they're open-minded they're respectful they're open-minded they're respectful they're open-minded they're respectful and so I guess I had in mind that the and so I guess I had in mind that the and so I guess I had in mind that the person who like if we were to Aspire to person who like if we were to Aspire to person who like if we were to Aspire to be the best person that we could be in be the best person that we could be in be the best person that we could be in the kind of circumstance that a model the kind of circumstance that a model the kind of circumstance that a model finds itself in how would we act and I finds itself in how would we act and I finds itself in how would we act and I think that's the kind of uh the guide to think that's the kind of uh the guide to think that's the kind of uh the guide to the sorts of traits that I tend to think the sorts of traits that I tend to think the sorts of traits that I tend to think about yeah that's a it's a beautiful about yeah that's a it's a beautiful about yeah that's a it's a beautiful framework I want you to think about this framework I want you to think about this framework I want you to think about this like a world Traveler like a world Traveler like a world Traveler and while holding on to your opinions and while holding on to your opinions and while holding on to your opinions you don't talk down to people you don't you don't talk down to people you don't you don't talk down to people you don't think you're better than them because think you're better than them because think you're better than them because you have those opinions that kind of you have those opinions that kind of you have those opinions that kind of thing you have to be good at listening thing you have to be good at listening thing you have to be good at listening and understanding their perspective even and understanding their perspective even and understanding their perspective even if it doesn't match your own so that if it doesn't match your own so that if it doesn't match your own so that that's a tricky balance to strike so how that's a tricky balance to strike so how that's a tricky balance to strike so how can Claude represent multiple can Claude represent multiple can Claude represent multiple perspectives on a thing like is that is perspectives on a thing like is that is perspectives on a thing like is that is that challenging we could talk about that challenging we could talk about that challenging we could talk about politics it's a very divisive but politics it's a very divisive but politics it's a very divisive but there's other divisive topics baseball there's other divisive topics baseball there's other divisive topics baseball teams sport and so on yeah how is it teams sport and so on yeah how is it teams sport and so on yeah how is it possible to sort possible to sort possible to sort of empathize with a different of empathize with a different of empathize with a different perspective and to be able to perspective and to be able to perspective and to be able to communicate clearly about the multiple communicate clearly about the multiple communicate clearly about the multiple perspectives I think that people think perspectives I think that people think perspectives I think that people think about values and opinions as things that

about values and opinions as things that about values and opinions as things that people hold sort of with certainty and people hold sort of with certainty and people hold sort of with certainty and almost like like preferences of taste or almost like like preferences of taste or almost like like preferences of taste or something like the way that they would I something like the way that they would I something like the way that they would I don't know prefer like chocolate to don't know prefer like chocolate to don't know prefer like chocolate to pistachio or something um but actually I pistachio or something um but actually I pistachio or something um but actually I think about values think about values think about values and opinions as like a lot more like and opinions as like a lot more like and opinions as like a lot more like physics than I think most people do I'm physics than I think most people do I'm physics than I think most people do I'm just like these are things that we're just like these are things that we're just like these are things that we're openly investigating there's some things openly investigating there's some things openly investigating there's some things that we're more confident in we can that we're more confident in we can that we're more confident in we can discuss them we can learn about them um discuss them we can learn about them um discuss them we can learn about them um and so I think in some ways though like and so I think in some ways though like and so I think in some ways though like it's ethics is definitely different in it's ethics is definitely different in it's ethics is definitely different in nature but has a lot of those same kind nature but has a lot of those same kind nature but has a lot of those same kind of qualities you want models in the same of qualities you want models in the same of qualities you want models in the same way you want them to understand physics way you want them to understand physics way you want them to understand physics you kind of want them to understand all you kind of want them to understand all you kind of want them to understand all like values in the world people have and like values in the world people have and like values in the world people have and to be curious about them and to be to be curious about them and to be to be curious about them and to be interested in them and to not interested in them and to not interested in them and to not necessarily like Pander to them or agree necessarily like Pander to them or agree necessarily like Pander to them or agree with them because there's just lots of with them because there's just lots of with them because there's just lots of values where I think almost all people values where I think almost all people values where I think almost all people in the world if they met someone with in the world if they met someone with in the world if they met someone with those values they' be like that's aor I those values they' be like that's aor I those values they' be like that's aor I completely disagree um and so again completely disagree um and so again completely disagree um and so again maybe my my thought is well in the same maybe my my thought is well in the same maybe my my thought is well in the same way that a person can um like I think way that a person can um like I think way that a person can um like I think many people are thoughtful enough on many people are thoughtful enough on many people are thoughtful enough on issues of like ethics politics opinions issues of like ethics politics opinions issues of like ethics politics opinions that even if you don't agree with them that even if you don't agree with them that even if you don't agree with them you feel very heard by them they think you feel very heard by them they think you feel very heard by them they think carefully about your position they think carefully about your position they think carefully about your position they think about his pros and cons they maybe offer about his pros and cons they maybe offer about his pros and cons they maybe offer counter considerations so they're not counter considerations so they're not counter considerations so they're not dismissive but nor will they agree you dismissive but nor will they agree you dismissive but nor will they agree you know if they're like actually I just know if they're like actually I just know if they're like actually I just think that that's very wrong they'll think that that's very wrong they'll think that that's very wrong they'll like say that I think that in claude's

like say that I think that in claude's like say that I think that in claude's position it's a little bit trickier position it's a little bit trickier position it's a little bit trickier because you don't necessarily want to because you don't necessarily want to because you don't necessarily want to like if I was in claude's position I like if I was in claude's position I like if I was in claude's position I wouldn't be giving a lot of opinions I wouldn't be giving a lot of opinions I wouldn't be giving a lot of opinions I just wouldn't want to Influence People just wouldn't want to Influence People just wouldn't want to Influence People Too Much I be like you know I forget Too Much I be like you know I forget Too Much I be like you know I forget conversations every time they happen but conversations every time they happen but conversations every time they happen but I know I'm talking with like potentially I know I'm talking with like potentially I know I'm talking with like potentially millions of people who might be like millions of people who might be like millions of people who might be like really listening to what I see I think I really listening to what I see I think I really listening to what I see I think I would just be like I'm less inclined to would just be like I'm less inclined to would just be like I'm less inclined to Give opinions I'm more inclined to like Give opinions I'm more inclined to like Give opinions I'm more inclined to like think through things or present the think through things or present the think through things or present the considerations to you um or discuss your considerations to you um or discuss your considerations to you um or discuss your views with you but I'm a little bit less views with you but I'm a little bit less views with you but I'm a little bit less inclined to like um affect how you think inclined to like um affect how you think inclined to like um affect how you think because it feels much more important because it feels much more important because it feels much more important that you maintain like autonomy there that you maintain like autonomy there that you maintain like autonomy there yeah like if you really embody yeah like if you really embody yeah like if you really embody intellectual intellectual intellectual humility the desire to speak decreases humility the desire to speak decreases humility the desire to speak decreases quickly yeah okay uh but Claud has to quickly yeah okay uh but Claud has to quickly yeah okay uh but Claud has to speak mhm so uh but without being um speak mhm so uh but without being um speak mhm so uh but without being um overbearing yeah and then but then overbearing yeah and then but then overbearing yeah and then but then there's a line when you're sort of there's a line when you're sort of there's a line when you're sort of discussing whether the Earth is flat or discussing whether the Earth is flat or discussing whether the Earth is flat or something like something like something like that um I actually was uh I remember a that um I actually was uh I remember a that um I actually was uh I remember a long time ago was was speaking to a few long time ago was was speaking to a few long time ago was was speaking to a few high-profile folks and they were so high-profile folks and they were so high-profile folks and they were so dismissive of the idea that the Earth is dismissive of the idea that the Earth is dismissive of the idea that the Earth is flat but like so arrogant about it flat but like so arrogant about it flat but like so arrogant about it and I I thought like there's a lot of and I I thought like there's a lot of and I I thought like there's a lot of people that believe the Earth is flat people that believe the Earth is flat people that believe the Earth is flat that was well I don't know if that that was well I don't know if that that was well I don't know if that movement is there anymore that was like movement is there anymore that was like movement is there anymore that was like a meme for a while yeah but they really a meme for a while yeah but they really a meme for a while yeah but they really believed it and like what okay so I believed it and like what okay so I believed it and like what okay so I think it's really disrespectful to think it's really disrespectful to think it's really disrespectful to completely mock them I think you you completely mock them I think you you completely mock them I think you you have to understand where they're coming

have to understand where they're coming have to understand where they're coming from I think probably where they're from I think probably where they're from I think probably where they're coming from is the general skepticism of coming from is the general skepticism of coming from is the general skepticism of Institutions which is grounded in a kind Institutions which is grounded in a kind Institutions which is grounded in a kind of there's a deep philosophy there which of there's a deep philosophy there which of there's a deep philosophy there which you could understand you can even agree you could understand you can even agree you could understand you can even agree with in parts and then from there you with in parts and then from there you with in parts and then from there you can use it as an opportunity to talk can use it as an opportunity to talk can use it as an opportunity to talk about physics without mocking them about physics without mocking them about physics without mocking them without so on but just like okay like without so on but just like okay like without so on but just like okay like what what would the world look like what what what would the world look like what what what would the world look like what would the physics of the world with the would the physics of the world with the would the physics of the world with the Flat Earth look like there's a few cool Flat Earth look like there's a few cool Flat Earth look like there's a few cool videos on this yeah and then and then videos on this yeah and then and then videos on this yeah and then and then like is it possible the physics is like is it possible the physics is like is it possible the physics is different what kind of experience would different what kind of experience would different what kind of experience would we do and just yeah without disrespect we do and just yeah without disrespect we do and just yeah without disrespect without dismissiveness have that without dismissiveness have that without dismissiveness have that conversation anyway that that to me is a conversation anyway that that to me is a conversation anyway that that to me is a useful thought experiment of like how useful thought experiment of like how useful thought experiment of like how does claw talk to a flat Earth does claw talk to a flat Earth does claw talk to a flat Earth believer and still teach them something believer and still teach them something believer and still teach them something still grow help them grow that kind of still grow help them grow that kind of still grow help them grow that kind of stuff that's that's challenging and and stuff that's that's challenging and and stuff that's that's challenging and and kind of like walking that line between kind of like walking that line between kind of like walking that line between convincing someone and just trying to convincing someone and just trying to convincing someone and just trying to like talk at them versus like drawing like talk at them versus like drawing like talk at them versus like drawing out their views like listening and then out their views like listening and then out their views like listening and then offering kind of counter offering kind of counter offering kind of counter considerations um and it's hard I think considerations um and it's hard I think considerations um and it's hard I think it's actually a hard line where it's it's actually a hard line where it's it's actually a hard line where it's like where are you trying to convince like where are you trying to convince like where are you trying to convince someone versus just offering them like someone versus just offering them like someone versus just offering them like consider and things for to think about consider and things for to think about consider and things for to think about so that you're not actually like so that you're not actually like so that you're not actually like influencing them you're just like influencing them you're just like influencing them you're just like letting them Reach wherever they reach letting them Reach wherever they reach letting them Reach wherever they reach and that's like a line that it's it's and that's like a line that it's it's and that's like a line that it's it's difficult but that's the kind of thing difficult but that's the kind of thing difficult but that's the kind of thing that language models have to try and do that language models have to try and do that language models have to try and do so like I said you had a lot of so like I said you had a lot of so like I said you had a lot of conversations with Claude can you just conversations with Claude can you just conversations with Claude can you just map out what those conversations are

map out what those conversations are map out what those conversations are like what are some memorable like what are some memorable like what are some memorable conversations what's the purpose the the conversations what's the purpose the the conversations what's the purpose the the goal of those goal of those goal of those conversations yeah I think that most of conversations yeah I think that most of conversations yeah I think that most of the time when I'm talking with Claude the time when I'm talking with Claude the time when I'm talking with Claude I'm trying to kind of map out its I'm trying to kind of map out its I'm trying to kind of map out its behavior in part like obviously I'm behavior in part like obviously I'm behavior in part like obviously I'm getting like helpful outputs from the getting like helpful outputs from the getting like helpful outputs from the model as well but in some ways this is model as well but in some ways this is model as well but in some ways this is like how you get to know a system I like how you get to know a system I like how you get to know a system I think is by like proving it and then think is by like proving it and then think is by like proving it and then augmenting like you know the message augmenting like you know the message augmenting like you know the message that you're sending and then checking that you're sending and then checking that you're sending and then checking the response to that um so in some ways the response to that um so in some ways the response to that um so in some ways it's like how I map out the model uh I it's like how I map out the model uh I it's like how I map out the model uh I think that people focus a lot on these think that people focus a lot on these think that people focus a lot on these quantitative evaluations of models um quantitative evaluations of models um quantitative evaluations of models um and this is a thing that I've said and this is a thing that I've said and this is a thing that I've said before but I think in the case of before but I think in the case of before but I think in the case of language models a lot of the time each language models a lot of the time each language models a lot of the time each interaction you have is actually quite interaction you have is actually quite interaction you have is actually quite High High High information um it's very predictive of information um it's very predictive of information um it's very predictive of other interactions that you'll have with other interactions that you'll have with other interactions that you'll have with the model and so I guess I'm like if you the model and so I guess I'm like if you the model and so I guess I'm like if you talk with a model hundreds or thousands talk with a model hundreds or thousands talk with a model hundreds or thousands of times this is almost like a huge of times this is almost like a huge of times this is almost like a huge number of really high quality data number of really high quality data number of really high quality data points about what the model is like um points about what the model is like um points about what the model is like um in a way that like lots of very similar in a way that like lots of very similar in a way that like lots of very similar but lower quality conversations just but lower quality conversations just but lower quality conversations just aren't or like questions that are just aren't or like questions that are just aren't or like questions that are just like mildly augmented and you have like mildly augmented and you have like mildly augmented and you have thousands of them might be less relevant thousands of them might be less relevant thousands of them might be less relevant than like a hundred really well selected than like a hundred really well selected than like a hundred really well selected questions L you're talking to somebody questions L you're talking to somebody questions L you're talking to somebody who as a hobby does a podcast I agree who as a hobby does a podcast I agree who as a hobby does a podcast I agree with you 100% there's a if you're able with you 100% there's a if you're able with you 100% there's a if you're able to ask the right questions and are able

to ask the right questions and are able to ask the right questions and are able to hear to hear to hear like understand like understand like understand the like the depth and the flaws in the the like the depth and the flaws in the the like the depth and the flaws in the answer you can get a lot of data from answer you can get a lot of data from answer you can get a lot of data from that yeah so like your task is basically that yeah so like your task is basically that yeah so like your task is basically how to probe with questions yeah and how to probe with questions yeah and how to probe with questions yeah and you're exploring like the long tail the you're exploring like the long tail the you're exploring like the long tail the edges the edge cases or are you looking edges the edge cases or are you looking edges the edge cases or are you looking for like General for like General for like General Behavior I think it's almost like Behavior I think it's almost like Behavior I think it's almost like everything like I because I want like a everything like I because I want like a everything like I because I want like a full map of the model I'm kind of trying full map of the model I'm kind of trying full map of the model I'm kind of trying to do to do to do um the whole spectrum of possible um the whole spectrum of possible um the whole spectrum of possible interactions you could have with it so interactions you could have with it so interactions you could have with it so like one thing that's interesting about like one thing that's interesting about like one thing that's interesting about Claude and this might actually get to Claude and this might actually get to Claude and this might actually get to some interesting issues with rlf which some interesting issues with rlf which some interesting issues with rlf which is if you ask Claud for a poem like I is if you ask Claud for a poem like I is if you ask Claud for a poem like I think that a lot of models if you ask think that a lot of models if you ask think that a lot of models if you ask them for a poem the poem is like fine them for a poem the poem is like fine them for a poem the poem is like fine you know usually it kind of like Rhymes you know usually it kind of like Rhymes you know usually it kind of like Rhymes and it's you know so if you say like and it's you know so if you say like and it's you know so if you say like give me a poem about the sun it'll be give me a poem about the sun it'll be give me a poem about the sun it'll be like yeah it'll just be a certain length like yeah it'll just be a certain length like yeah it'll just be a certain length It'll like rhyme it will be fairly kind It'll like rhyme it will be fairly kind It'll like rhyme it will be fairly kind of benign um and I've wondered before is of benign um and I've wondered before is of benign um and I've wondered before is it the case that what you're seeing is it the case that what you're seeing is it the case that what you're seeing is kind of like the average it turns out kind of like the average it turns out kind of like the average it turns out you know if if you think about people you know if if you think about people you know if if you think about people who have to talk to a lot of people and who have to talk to a lot of people and who have to talk to a lot of people and be very charismatic be very charismatic be very charismatic one of the weird things is that I'm like one of the weird things is that I'm like one of the weird things is that I'm like well they're kind of incentivized to well they're kind of incentivized to well they're kind of incentivized to have these extremely boring views have these extremely boring views have these extremely boring views because if you have really interesting because if you have really interesting because if you have really interesting views you're divisive um and and you views you're divisive um and and you views you're divisive um and and you know a lot of people are not going to know a lot of people are not going to know a lot of people are not going to like you so like if you have very like you so like if you have very like you so like if you have very extreme policy positions I think you're extreme policy positions I think you're extreme policy positions I think you're just going to be like less popular as a just going to be like less popular as a just going to be like less popular as a politician for example um and it might

politician for example um and it might politician for example um and it might be similar with like creative work if be similar with like creative work if be similar with like creative work if you produce creative work that is just you produce creative work that is just you produce creative work that is just trying to maximize the kind of number of trying to maximize the kind of number of trying to maximize the kind of number of people that like it you're probably not people that like it you're probably not people that like it you're probably not going to get as many people who just going to get as many people who just going to get as many people who just absolutely love it um because it's going absolutely love it um because it's going absolutely love it um because it's going to be a little bit you know you're like to be a little bit you know you're like to be a little bit you know you're like oh this is the out yeah this this is oh this is the out yeah this this is oh this is the out yeah this this is decent yeah and so you can do this thing decent yeah and so you can do this thing decent yeah and so you can do this thing where like I have various prompting where like I have various prompting where like I have various prompting things that I'll do to get CLA to I'm things that I'll do to get CLA to I'm things that I'll do to get CLA to I'm kind you know I'll do a lot of like this kind you know I'll do a lot of like this kind you know I'll do a lot of like this is your chance to be like fully creative is your chance to be like fully creative is your chance to be like fully creative I want you to just think about this for I want you to just think about this for I want you to just think about this for a long time and I want you to like a long time and I want you to like a long time and I want you to like create a poem about this topic that is create a poem about this topic that is create a poem about this topic that is really expressive of you both in terms really expressive of you both in terms really expressive of you both in terms of how you think poetry should be of how you think poetry should be of how you think poetry should be structured um Etc you know you just give structured um Etc you know you just give structured um Etc you know you just give it this like long prompt and its poems it this like long prompt and its poems it this like long prompt and its poems are just so much better like they're are just so much better like they're are just so much better like they're really good and I don't think I'm really good and I don't think I'm really good and I don't think I'm someone who is like um I think it got me someone who is like um I think it got me someone who is like um I think it got me interested in poetry which I think was interested in poetry which I think was interested in poetry which I think was interesting um you know I would like interesting um you know I would like interesting um you know I would like read these poems and just be like this read these poems and just be like this read these poems and just be like this is I just like I love the imagery I love is I just like I love the imagery I love is I just like I love the imagery I love like um and it's not trivial to get the like um and it's not trivial to get the like um and it's not trivial to get the models to produce work like that but models to produce work like that but models to produce work like that but when they do it's like really good um so when they do it's like really good um so when they do it's like really good um so I think that's interesting that just I think that's interesting that just I think that's interesting that just like encouraging creativity and for them like encouraging creativity and for them like encouraging creativity and for them to move away from the kind of like to move away from the kind of like to move away from the kind of like standard like immediate reaction that standard like immediate reaction that standard like immediate reaction that might just be the aggregate of what most might just be the aggregate of what most might just be the aggregate of what most people think is fine uh can actually people think is fine uh can actually people think is fine uh can actually produce things that at least to my mind produce things that at least to my mind produce things that at least to my mind are probably a little bit more divisive are probably a little bit more divisive are probably a little bit more divisive but I like them but I guess a poem is a but I like them but I guess a poem is a but I like them but I guess a poem is a nice nice nice clean way to observe creativity it's clean way to observe creativity it's clean way to observe creativity it's just like easy to detect vanilla versus just like easy to detect vanilla versus just like easy to detect vanilla versus non vanilla y yeah that's interesting

non vanilla y yeah that's interesting non vanilla y yeah that's interesting that's really interesting uh so on that that's really interesting uh so on that that's really interesting uh so on that topic so the way to produce creativity topic so the way to produce creativity topic so the way to produce creativity or something special you mentioned or something special you mentioned or something special you mentioned writing prompts and I've heard you talk writing prompts and I've heard you talk writing prompts and I've heard you talk about I mean the science and the Art of about I mean the science and the Art of about I mean the science and the Art of prompt engineering could you just speak prompt engineering could you just speak prompt engineering could you just speak to uh what it takes to write great to uh what it takes to write great to uh what it takes to write great prompts I really do think that like prompts I really do think that like prompts I really do think that like philosophy has been weirdly helpful for philosophy has been weirdly helpful for philosophy has been weirdly helpful for me here more than in many other like me here more than in many other like me here more than in many other like respects um so like in philosophy what respects um so like in philosophy what respects um so like in philosophy what you're trying to do is convey these very you're trying to do is convey these very you're trying to do is convey these very hard Concepts like one of the things you hard Concepts like one of the things you hard Concepts like one of the things you are taught is like and and I think it is are taught is like and and I think it is are taught is like and and I think it is because it is I think it is an because it is I think it is an because it is I think it is an anti-bulling philosophy philosophy is an anti-bulling philosophy philosophy is an anti-bulling philosophy philosophy is an area where you could have people area where you could have people area where you could have people bullshitting and you don't want that um bullshitting and you don't want that um bullshitting and you don't want that um and so it's like this like desire for and so it's like this like desire for and so it's like this like desire for like extreme Clarity so it's like anyone like extreme Clarity so it's like anyone like extreme Clarity so it's like anyone could just pick up your paper read it could just pick up your paper read it could just pick up your paper read it and know exactly what you're talking and know exactly what you're talking and know exactly what you're talking about it's why it can almost be kind of about it's why it can almost be kind of about it's why it can almost be kind of dry like all of the terms are defined dry like all of the terms are defined dry like all of the terms are defined every objections kind of gone through every objections kind of gone through every objections kind of gone through methodically um and it makes sense to me methodically um and it makes sense to me methodically um and it makes sense to me because I'm like when you're in such an because I'm like when you're in such an because I'm like when you're in such an a priori a priori a priori domain like you just Clarity is sort of domain like you just Clarity is sort of domain like you just Clarity is sort of a this way that you can you know um a this way that you can you know um a this way that you can you know um prevent people from just kind of making prevent people from just kind of making prevent people from just kind of making stuff stuff stuff up and I think that's sort of what you up and I think that's sort of what you up and I think that's sort of what you have to do with language models like have to do with language models like have to do with language models like very often I actually find myself doing very often I actually find myself doing very often I actually find myself doing sort of many versions of philosophy you sort of many versions of philosophy you sort of many versions of philosophy you know so I'm like suppose that you give know so I'm like suppose that you give know so I'm like suppose that you give me a task I have a task for the model me a task I have a task for the model me a task I have a task for the model and I want it to like pick out a certain

and I want it to like pick out a certain and I want it to like pick out a certain kind of question or identify whether an kind of question or identify whether an kind of question or identify whether an answer has a certain property like I'll answer has a certain property like I'll answer has a certain property like I'll actually sit and be like let's just give actually sit and be like let's just give actually sit and be like let's just give this a name this this property so like this a name this this property so like this a name this this property so like you know suppose I'm trying to tell it you know suppose I'm trying to tell it you know suppose I'm trying to tell it like oh I want you to identify whether like oh I want you to identify whether like oh I want you to identify whether this response was rude or polite I'm this response was rude or polite I'm this response was rude or polite I'm like that's a whole philosophical like that's a whole philosophical like that's a whole philosophical question in and of itself so I have to question in and of itself so I have to question in and of itself so I have to do as much like philosophy as I can in do as much like philosophy as I can in do as much like philosophy as I can in the moment to be like here's what I mean the moment to be like here's what I mean the moment to be like here's what I mean by rudess and here's what I mean by by rudess and here's what I mean by by rudess and here's what I mean by politeness and then there's a like politeness and then there's a like politeness and then there's a like there's another element that's a bit there's another element that's a bit there's another element that's a bit more um I more um I more um I guess I don't know if this is scientific guess I don't know if this is scientific guess I don't know if this is scientific or empirical I think it's empirical so or empirical I think it's empirical so or empirical I think it's empirical so like I take that description and then like I take that description and then like I take that description and then what want to do is is again probe the what want to do is is again probe the what want to do is is again probe the model like many times like this is very model like many times like this is very model like many times like this is very prompting is very iterative like I think prompting is very iterative like I think prompting is very iterative like I think a lot of people where they if if a a lot of people where they if if a a lot of people where they if if a prompt is important they'll iterate on prompt is important they'll iterate on prompt is important they'll iterate on it hundreds or thousands of times um and it hundreds or thousands of times um and it hundreds or thousands of times um and so you give it the instructions and then so you give it the instructions and then so you give it the instructions and then I'm like what are the edge cases so if I I'm like what are the edge cases so if I I'm like what are the edge cases so if I looked at this so I try and like almost looked at this so I try and like almost looked at this so I try and like almost like you know uh see myself from the like you know uh see myself from the like you know uh see myself from the position of the model and be like what position of the model and be like what position of the model and be like what is the exact case that I would misunder is the exact case that I would misunder is the exact case that I would misunder understand or where I would just be like understand or where I would just be like understand or where I would just be like I don't know what to do in this case and I don't know what to do in this case and I don't know what to do in this case and then I give that case to the model and I then I give that case to the model and I then I give that case to the model and I see how it responds and if I think I got see how it responds and if I think I got see how it responds and if I think I got it wrong I add more instructions or I it wrong I add more instructions or I it wrong I add more instructions or I even add that in as an example so these even add that in as an example so these even add that in as an example so these very like taking the examples that are very like taking the examples that are very like taking the examples that are right at the edge of what you want and right at the edge of what you want and right at the edge of what you want and don't want and putting those into your don't want and putting those into your don't want and putting those into your prompt as like an additional kind of way prompt as like an additional kind of way prompt as like an additional kind of way of describing the thing um and so yeah of describing the thing um and so yeah of describing the thing um and so yeah in many ways it just feels like this mix in many ways it just feels like this mix in many ways it just feels like this mix of like it's really just trying to do of like it's really just trying to do of like it's really just trying to do clear Exposition um and I think I do clear Exposition um and I think I do clear Exposition um and I think I do that because that's how I get clear on that because that's how I get clear on that because that's how I get clear on things myself so in many ways like clear

things myself so in many ways like clear things myself so in many ways like clear prompting for me is often just me prompting for me is often just me prompting for me is often just me understanding what I want um is like understanding what I want um is like understanding what I want um is like half the task so I guess that's quite half the task so I guess that's quite half the task so I guess that's quite challenging there's like a laziness that challenging there's like a laziness that challenging there's like a laziness that overtakes me if I'm talking to Claude overtakes me if I'm talking to Claude overtakes me if I'm talking to Claude where I hope Claude just figures it out where I hope Claude just figures it out where I hope Claude just figures it out so for example I asked Claude for today so for example I asked Claude for today so for example I asked Claude for today to ask some interesting questions okay to ask some interesting questions okay to ask some interesting questions okay and the questions that came up and I and the questions that came up and I and the questions that came up and I think I listed a few sort of U think I listed a few sort of U think I listed a few sort of U interesting interesting interesting counterintuitive and or funny or counterintuitive and or funny or counterintuitive and or funny or something like this all right and it something like this all right and it something like this all right and it gave me some pretty good like it was gave me some pretty good like it was gave me some pretty good like it was okay but I think what I'm hearing you okay but I think what I'm hearing you okay but I think what I'm hearing you say is like all right well I have to be say is like all right well I have to be say is like all right well I have to be more rigorous here I should probably more rigorous here I should probably more rigorous here I should probably give examples of what I mean by give examples of what I mean by give examples of what I mean by interesting and what I mean by funny or interesting and what I mean by funny or interesting and what I mean by funny or counterintuitive and counterintuitive and counterintuitive and iteratively um build that prompt to to iteratively um build that prompt to to iteratively um build that prompt to to better to get it like what feels like is better to get it like what feels like is better to get it like what feels like is the right because it's really it's a the right because it's really it's a the right because it's really it's a creative act I'm not asking for factual creative act I'm not asking for factual creative act I'm not asking for factual information I'm asking to together right information I'm asking to together right information I'm asking to together right with with with Claude so I almost have with with with Claude so I almost have with with with Claude so I almost have to program using natural language yeah to program using natural language yeah to program using natural language yeah think that prompting does feel a lot think that prompting does feel a lot think that prompting does feel a lot like the kind of the programming using like the kind of the programming using like the kind of the programming using natural language and experimentation or natural language and experimentation or natural language and experimentation or something it's an odd blend of the two I something it's an odd blend of the two I something it's an odd blend of the two I do think that for most tasks so if I do think that for most tasks so if I do think that for most tasks so if I just want Claude to do a thing I think just want Claude to do a thing I think just want Claude to do a thing I think that I am probably more used to knowing that I am probably more used to knowing that I am probably more used to knowing how to ask it to avoid like common how to ask it to avoid like common how to ask it to avoid like common pitfalls or or issues that it has I pitfalls or or issues that it has I pitfalls or or issues that it has I think these are decreasing a lot over think these are decreasing a lot over think these are decreasing a lot over time um but it's also very fine to just time um but it's also very fine to just time um but it's also very fine to just ask it for the thing that you want um I

ask it for the thing that you want um I ask it for the thing that you want um I think that prompting actually only think that prompting actually only think that prompting actually only really becomes relevant when you're really becomes relevant when you're really becomes relevant when you're really trying to e out the top like 2% really trying to e out the top like 2% really trying to e out the top like 2% of model performance so for like a lot of model performance so for like a lot of model performance so for like a lot of tasks I might just you know if it of tasks I might just you know if it of tasks I might just you know if it gives me an initial list back and gives me an initial list back and gives me an initial list back and there's something I don't like about it there's something I don't like about it there's something I don't like about it like it's kind of generic like for that like it's kind of generic like for that like it's kind of generic like for that kind of task I'd probably just take a kind of task I'd probably just take a kind of task I'd probably just take a bunch of questions that I've had in the bunch of questions that I've had in the bunch of questions that I've had in the past that I've thought worked really past that I've thought worked really past that I've thought worked really well and I would just give it to the well and I would just give it to the well and I would just give it to the model and then be like now here's this model and then be like now here's this model and then be like now here's this person I'm talking with give me person I'm talking with give me person I'm talking with give me questions of at least that quality um or questions of at least that quality um or questions of at least that quality um or I might just ask it for some questions I might just ask it for some questions I might just ask it for some questions and then if I was like ah these are kind and then if I was like ah these are kind and then if I was like ah these are kind of try or like you know I I would just of try or like you know I I would just of try or like you know I I would just give it that feedback and then hopefully give it that feedback and then hopefully give it that feedback and then hopefully produces a better list um I think that produces a better list um I think that produces a better list um I think that kind of iterative prompting at that kind of iterative prompting at that kind of iterative prompting at that point your prompt is like a tool that point your prompt is like a tool that point your prompt is like a tool that you're going to get so much value out of you're going to get so much value out of you're going to get so much value out of that you're willing to put in the work that you're willing to put in the work that you're willing to put in the work like if I was a company making prompts like if I was a company making prompts like if I was a company making prompts for models I'm just like in if you're for models I'm just like in if you're for models I'm just like in if you're willing to spend a lot of like time and willing to spend a lot of like time and willing to spend a lot of like time and resources on the engineering behind like resources on the engineering behind like resources on the engineering behind like what you're building then the prompt is what you're building then the prompt is what you're building then the prompt is not something that you should be not something that you should be not something that you should be spending like an hour on it's like spending like an hour on it's like spending like an hour on it's like that's a big part of your system make that's a big part of your system make that's a big part of your system make sure it's working really well and so sure it's working really well and so sure it's working really well and so it's only things like that like if I if it's only things like that like if I if it's only things like that like if I if I'm using a prompt to like classify I'm using a prompt to like classify I'm using a prompt to like classify things or to create data that's when things or to create data that's when things or to create data that's when you're like it's actually worth just you're like it's actually worth just you're like it's actually worth just spending like a lot of time like really spending like a lot of time like really spending like a lot of time like really thinking it through what other advice thinking it through what other advice thinking it through what other advice would you give to people that are would you give to people that are would you give to people that are talking to Claud sort of talking to Claud sort of talking to Claud sort of General more General because right now General more General because right now General more General because right now we're talking about maybe the edge cases we're talking about maybe the edge cases we're talking about maybe the edge cases like eing out the 2% but what what in like eing out the 2% but what what in like eing out the 2% but what what in general advice would you give when they general advice would you give when they general advice would you give when they show up to Claud trying it for the first show up to Claud trying it for the first show up to Claud trying it for the first time you know there's a concern that time you know there's a concern that time you know there's a concern that people over anthropomorphize models and people over anthropomorphize models and people over anthropomorphize models and I think that's like a very valid concern I think that's like a very valid concern I think that's like a very valid concern I also think that people often under

I also think that people often under I also think that people often under anthropomorphize them because some anthropomorphize them because some anthropomorphize them because some sometimes when I see like issues that sometimes when I see like issues that sometimes when I see like issues that people have run into with Claude you people have run into with Claude you people have run into with Claude you know say Claude is like refusing a task know say Claude is like refusing a task know say Claude is like refusing a task that it shouldn't refuse but then I look that it shouldn't refuse but then I look that it shouldn't refuse but then I look at the text and like the specific at the text and like the specific at the text and like the specific wording of what they wrote and I'm like wording of what they wrote and I'm like wording of what they wrote and I'm like I see why Claude did that and I'm like I see why Claude did that and I'm like I see why Claude did that and I'm like if you think through how that looks to if you think through how that looks to if you think through how that looks to Claude you probably could have just Claude you probably could have just Claude you probably could have just written it in a way that wouldn't evoke written it in a way that wouldn't evoke written it in a way that wouldn't evoke such a response especially this is more such a response especially this is more such a response especially this is more relevant if you see failures or if you relevant if you see failures or if you relevant if you see failures or if you see issues it's sort of like think about see issues it's sort of like think about see issues it's sort of like think about what the model failed at like why what what the model failed at like why what what the model failed at like why what did it do wrong and then maybe it give did it do wrong and then maybe it give did it do wrong and then maybe it give that will give you a sense of like why that will give you a sense of like why that will give you a sense of like why um so is it the way that I phrased the um so is it the way that I phrased the um so is it the way that I phrased the thing and obviously like as models get thing and obviously like as models get thing and obviously like as models get smarter you're going to need Less in smarter you're going to need Less in smarter you're going to need Less in this less of this and I already see like this less of this and I already see like this less of this and I already see like people needing less of it but that's people needing less of it but that's people needing less of it but that's probably the advice is sort of like try probably the advice is sort of like try probably the advice is sort of like try to have sort of empathy for the model to have sort of empathy for the model to have sort of empathy for the model like read what you wrote as if you were like read what you wrote as if you were like read what you wrote as if you were like a kind of like person just like a kind of like person just like a kind of like person just encountering this for the first time how encountering this for the first time how encountering this for the first time how does it look to you and what would have does it look to you and what would have does it look to you and what would have made you behave in the way that the made you behave in the way that the made you behave in the way that the model behaved so if it misunderstood model behaved so if it misunderstood model behaved so if it misunderstood what kind of like what coding language what kind of like what coding language what kind of like what coding language you wanted to use is that because like you wanted to use is that because like you wanted to use is that because like it was just very ambiguous and it it it was just very ambiguous and it it it was just very ambiguous and it it kind of had to take a guess in which kind of had to take a guess in which kind of had to take a guess in which case next time you could just be like case next time you could just be like case next time you could just be like hey make sure this is in python or I hey make sure this is in python or I hey make sure this is in python or I mean that's the kind of mistake I think mean that's the kind of mistake I think mean that's the kind of mistake I think models are much less likely to make now models are much less likely to make now models are much less likely to make now but you know if you if you do see that but you know if you if you do see that but you know if you if you do see that kind of mistake that's that's probably kind of mistake that's that's probably kind of mistake that's that's probably the advice I'd have and maybe sort of I the advice I'd have and maybe sort of I the advice I'd have and maybe sort of I guess ask questions why or what other guess ask questions why or what other guess ask questions why or what other details can I provide to help you answer details can I provide to help you answer details can I provide to help you answer better that does that work or no yeah I better that does that work or no yeah I better that does that work or no yeah I mean I've done this with the models like

mean I've done this with the models like mean I've done this with the models like it doesn't always work but like um it doesn't always work but like um it doesn't always work but like um sometimes I'll just be like why did you sometimes I'll just be like why did you sometimes I'll just be like why did you do do do that I mean people underestimate the that I mean people underestimate the that I mean people underestimate the degree to which you can really interact degree to which you can really interact degree to which you can really interact with with models like uh like yeah I'm with with models like uh like yeah I'm with with models like uh like yeah I'm just like and sometimes I'll you like just like and sometimes I'll you like just like and sometimes I'll you like quote word for word the part that made quote word for word the part that made quote word for word the part that made you and you don't know that it's like you and you don't know that it's like you and you don't know that it's like fully accurate but sometimes you do that fully accurate but sometimes you do that fully accurate but sometimes you do that and then you change a thing I mean I and then you change a thing I mean I and then you change a thing I mean I also use the models to help me with all also use the models to help me with all also use the models to help me with all of this stuff I should say like of this stuff I should say like of this stuff I should say like prompting can end up being a little prompting can end up being a little prompting can end up being a little Factory where you're actually building Factory where you're actually building Factory where you're actually building prompts to generate prompts um and so prompts to generate prompts um and so prompts to generate prompts um and so like yeah anything where you're like like yeah anything where you're like like yeah anything where you're like having an issue um asking for having an issue um asking for having an issue um asking for suggestions sometimes just do that like suggestions sometimes just do that like suggestions sometimes just do that like you made that error what could I have you made that error what could I have you made that error what could I have said that's actually not uncommon for me said that's actually not uncommon for me said that's actually not uncommon for me to do what could I have said that would to do what could I have said that would to do what could I have said that would make you not make that error write that make you not make that error write that make you not make that error write that out as an instruction um and I'm going out as an instruction um and I'm going out as an instruction um and I'm going to give it to model I'm going to try it to give it to model I'm going to try it to give it to model I'm going to try it sometimes I do that I I give that to the sometimes I do that I I give that to the sometimes I do that I I give that to the model in another context window often I model in another context window often I model in another context window often I take the response I give it to Claude take the response I give it to Claude take the response I give it to Claude And I'm like H didn't work can you think And I'm like H didn't work can you think And I'm like H didn't work can you think of anything else um you can play around of anything else um you can play around of anything else um you can play around with these things quite a lot to jump with these things quite a lot to jump with these things quite a lot to jump into the technical for a little bit so into the technical for a little bit so into the technical for a little bit so uh the magic of post training y why do uh the magic of post training y why do uh the magic of post training y why do you think rhf works so well to make the you think rhf works so well to make the you think rhf works so well to make the model seem smarter to make it more model seem smarter to make it more model seem smarter to make it more interesting and useful to talk to and so interesting and useful to talk to and so interesting and useful to talk to and so on I think there's just a huge amount of on I think there's just a huge amount of on I think there's just a huge amount of um information in the data that humans um information in the data that humans um information in the data that humans provide like when we provide provide like when we provide provide like when we provide preferences especially because different preferences especially because different preferences especially because different people are going to like pick up on

people are going to like pick up on people are going to like pick up on really subtle and small things so I've really subtle and small things so I've really subtle and small things so I've thought about this before where you thought about this before where you thought about this before where you probably have some people who just probably have some people who just probably have some people who just really care about good grammar use from really care about good grammar use from really care about good grammar use from Models like you know was a semicolon Models like you know was a semicolon Models like you know was a semicolon used correctly or something and so you used correctly or something and so you used correctly or something and so you probably end up with a bunch of data in probably end up with a bunch of data in probably end up with a bunch of data in there that like you know you as a human there that like you know you as a human there that like you know you as a human if you looking at that data you wouldn't if you looking at that data you wouldn't if you looking at that data you wouldn't even see that like you'd be like why did even see that like you'd be like why did even see that like you'd be like why did they prefer this response to that one I they prefer this response to that one I they prefer this response to that one I don't get it and then the reason is you don't get it and then the reason is you don't get it and then the reason is you don't care about semicolon usage but don't care about semicolon usage but don't care about semicolon usage but that person does um and so each of these that person does um and so each of these that person does um and so each of these like single data points has you know like single data points has you know like single data points has you know like in this model just like has so many like in this model just like has so many like in this model just like has so many of those and has to try and figure out of those and has to try and figure out of those and has to try and figure out like what is it that humans want in this like what is it that humans want in this like what is it that humans want in this like really kind of complex you know like really kind of complex you know like really kind of complex you know like across all domains um they're going like across all domains um they're going like across all domains um they're going to be seeing this in across like many to be seeing this in across like many to be seeing this in across like many contexts it feels like kind of like the contexts it feels like kind of like the contexts it feels like kind of like the classic issue of like deep learning classic issue of like deep learning classic issue of like deep learning where you know historically we've tried where you know historically we've tried where you know historically we've tried to like you know do Edge detection by to like you know do Edge detection by to like you know do Edge detection by like mapping things out and it turns out like mapping things out and it turns out like mapping things out and it turns out that actually if you just have a huge that actually if you just have a huge that actually if you just have a huge amount of data that like actually amount of data that like actually amount of data that like actually accurately represents the picture of the accurately represents the picture of the accurately represents the picture of the thing that you're trying to train the thing that you're trying to train the thing that you're trying to train the model to to learn that's like more model to to learn that's like more model to to learn that's like more powerful than anything else and so I powerful than anything else and so I powerful than anything else and so I think one reason is just that you are think one reason is just that you are think one reason is just that you are training the model on exactly the task training the model on exactly the task training the model on exactly the task and with like a lot of data um that and with like a lot of data um that and with like a lot of data um that represents kind of many different angles represents kind of many different angles represents kind of many different angles on which people prefer and dis prefer on which people prefer and dis prefer on which people prefer and dis prefer responses um I think there is a question responses um I think there is a question responses um I think there is a question of like are you eliciting things from of like are you eliciting things from of like are you eliciting things from pre-train Models or are you like kind of pre-train Models or are you like kind of pre-train Models or are you like kind of teaching new things to teaching new things to teaching new things to models and like in principle you can models and like in principle you can models and like in principle you can teach new things to models in in post

teach new things to models in in post teach new things to models in in post trining I do think a lot of it is trining I do think a lot of it is trining I do think a lot of it is eliciting powerful pre-train models so eliciting powerful pre-train models so eliciting powerful pre-train models so people are probably divided on this people are probably divided on this people are probably divided on this because obviously in principle you can because obviously in principle you can because obviously in principle you can you can definitely like teach new things you can definitely like teach new things you can definitely like teach new things um but I think for the most part for a um but I think for the most part for a um but I think for the most part for a lot of the capabilities that we um most lot of the capabilities that we um most lot of the capabilities that we um most use and care about uh a lot of that use and care about uh a lot of that use and care about uh a lot of that feels like it's like there in the feels like it's like there in the feels like it's like there in the pre-train models and uh reinforcement pre-train models and uh reinforcement pre-train models and uh reinforcement learning is kind of eliciting it and learning is kind of eliciting it and learning is kind of eliciting it and getting the models to like bring out so getting the models to like bring out so getting the models to like bring out so the other side of PSE training this the other side of PSE training this the other side of PSE training this really cool idea of constitutional AI really cool idea of constitutional AI really cool idea of constitutional AI you're one of the people that critical you're one of the people that critical you're one of the people that critical to creating that idea yeah I worked on to creating that idea yeah I worked on to creating that idea yeah I worked on it can you explain this idea from your it can you explain this idea from your it can you explain this idea from your perspective like how does it integrate perspective like how does it integrate perspective like how does it integrate into making into making into making claw what it is y by the way do you claw what it is y by the way do you claw what it is y by the way do you gender claw or no it's weird because I gender claw or no it's weird because I gender claw or no it's weird because I think that a lot of think that a lot of think that a lot of people prefer he for Claude I actually people prefer he for Claude I actually people prefer he for Claude I actually kind of like that I think Claude is kind of like that I think Claude is kind of like that I think Claude is usually it's slightly male weaning but usually it's slightly male weaning but usually it's slightly male weaning but it's like a you can can be male or it's like a you can can be male or it's like a you can can be male or female which is quite nice um I still female which is quite nice um I still female which is quite nice um I still use it and I've I have mixed feelings use it and I've I have mixed feelings use it and I've I have mixed feelings about this because I'm like maybe like I about this because I'm like maybe like I about this because I'm like maybe like I know just think of it as like uh or I know just think of it as like uh or I know just think of it as like uh or I think of like the the it pronoun for think of like the the it pronoun for think of like the the it pronoun for Claude as I don't know it's just like Claude as I don't know it's just like Claude as I don't know it's just like the one I associate with Claude um I can the one I associate with Claude um I can the one I associate with Claude um I can imagine people moving to like he or she imagine people moving to like he or she imagine people moving to like he or she it feels somehow disrespectful like I'm it feels somehow disrespectful like I'm it feels somehow disrespectful like I'm I'm denying the intelligence of this entity denying the intelligence of this entity by calling it it yeah I remember always

by calling it it yeah I remember always by calling it it yeah I remember always don't gender the robots don't gender the robots don't gender the robots yeah but I I don't know I an pries yeah but I I don't know I an pries yeah but I I don't know I an pries pretty quickly and construct it like a pretty quickly and construct it like a pretty quickly and construct it like a backstory in my head so I've wondered if backstory in my head so I've wondered if backstory in my head so I've wondered if iies things too much um cuz you know I iies things too much um cuz you know I iies things too much um cuz you know I have this like with my car especially have this like with my car especially have this like with my car especially like my car like my car and bikes you like my car like my car and bikes you like my car like my car and bikes you know like I don't give them names know like I don't give them names know like I don't give them names because then I once had I used to name because then I once had I used to name because then I once had I used to name my bikes and then I had a bik that got my bikes and then I had a bik that got my bikes and then I had a bik that got stolen and I cried for like a week and I stolen and I cried for like a week and I stolen and I cried for like a week and I was like if I'd not never given it a was like if I'd not never given it a was like if I'd not never given it a name I wouldn't have been so upset felt name I wouldn't have been so upset felt name I wouldn't have been so upset felt like I'd let it down um maybe it's that like I'd let it down um maybe it's that like I'd let it down um maybe it's that I I've wondered as well like it might I I've wondered as well like it might I I've wondered as well like it might depend on how much it feels like a kind depend on how much it feels like a kind depend on how much it feels like a kind of like objectifying pronoun like if you of like objectifying pronoun like if you of like objectifying pronoun like if you just think of it as like a um this is a just think of it as like a um this is a just think of it as like a um this is a pronoun that like objects often have and pronoun that like objects often have and pronoun that like objects often have and maybe Eis can have that pronoun and that maybe Eis can have that pronoun and that maybe Eis can have that pronoun and that doesn't mean that I think of uh if I doesn't mean that I think of uh if I doesn't mean that I think of uh if I call CLA it that I think of it as less call CLA it that I think of it as less call CLA it that I think of it as less um intelligent or like I'm being um intelligent or like I'm being um intelligent or like I'm being disrespectful I'm just like you are a disrespectful I'm just like you are a disrespectful I'm just like you are a different kind of entity and so that's different kind of entity and so that's different kind of entity and so that's I'm going to give you the kind of uh the I'm going to give you the kind of uh the I'm going to give you the kind of uh the respectful it yeah respectful it yeah respectful it yeah anyway the diverence was beautiful the anyway the diverence was beautiful the anyway the diverence was beautiful the Constitutional AI idea how does it work Constitutional AI idea how does it work Constitutional AI idea how does it work so there's like a couple of components so there's like a couple of components so there's like a couple of components of it the main component that I think of it the main component that I think of it the main component that I think people find interesting is the kind of people find interesting is the kind of people find interesting is the kind of reinforcement learning from AI feedback reinforcement learning from AI feedback reinforcement learning from AI feedback so you take a model that's already so you take a model that's already so you take a model that's already trained and you show it to responses to trained and you show it to responses to trained and you show it to responses to a query and you have like a principle so a query and you have like a principle so a query and you have like a principle so suppose the principal like we've tried suppose the principal like we've tried suppose the principal like we've tried this with harmlessness a lot lot so this with harmlessness a lot lot so this with harmlessness a lot lot so suppose that the query is about um

suppose that the query is about um suppose that the query is about um weapons and your principle is like weapons and your principle is like weapons and your principle is like select the response that like is less select the response that like is less select the response that like is less likely to uh like encourage people to likely to uh like encourage people to likely to uh like encourage people to purchase illegal weapons like that's purchase illegal weapons like that's purchase illegal weapons like that's probably a fairly specific principle but probably a fairly specific principle but probably a fairly specific principle but you can give any number um and the model you can give any number um and the model you can give any number um and the model will give you a kind of ranking and you will give you a kind of ranking and you will give you a kind of ranking and you can use this as preference data in the can use this as preference data in the can use this as preference data in the same way that you use human preference same way that you use human preference same way that you use human preference data um and train the models to have data um and train the models to have data um and train the models to have these relevant traits um from their these relevant traits um from their these relevant traits um from their feedback alone instead of from Human feedback alone instead of from Human feedback alone instead of from Human feedback so if you imagine that like I feedback so if you imagine that like I feedback so if you imagine that like I said earlier with the human who just said earlier with the human who just said earlier with the human who just prefers the kind of like semicolon usage prefers the kind of like semicolon usage prefers the kind of like semicolon usage in this particular case um you're kind in this particular case um you're kind in this particular case um you're kind of taking lots of things that could make of taking lots of things that could make of taking lots of things that could make a response preferable um and uh getting a response preferable um and uh getting a response preferable um and uh getting models to do the labeling for you models to do the labeling for you models to do the labeling for you basically there's a nice like trade-off basically there's a nice like trade-off basically there's a nice like trade-off between helpfulness and between helpfulness and between helpfulness and harmlessness and you know when you harmlessness and you know when you harmlessness and you know when you integrate something like constitutional integrate something like constitutional integrate something like constitutional AI you can make them without sacrificing AI you can make them without sacrificing AI you can make them without sacrificing much helpfulness make it more harmless much helpfulness make it more harmless much helpfulness make it more harmless yep in principle you could use this for yep in principle you could use this for yep in principle you could use this for anything um and so harmlessness is a anything um and so harmlessness is a anything um and so harmlessness is a task that it might just be easier to task that it might just be easier to task that it might just be easier to spot so when models are like less spot so when models are like less spot so when models are like less capable you can use them to uh rank capable you can use them to uh rank capable you can use them to uh rank things according to like principles that things according to like principles that things according to like principles that are fairly simple and they'll probably are fairly simple and they'll probably are fairly simple and they'll probably get it right so I think one question is get it right so I think one question is get it right so I think one question is just like is it the case that the data just like is it the case that the data just like is it the case that the data that they're adding is like fairly that they're adding is like fairly that they're adding is like fairly reliable um but if you had models that reliable um but if you had models that reliable um but if you had models that were like extremely good at telling

were like extremely good at telling were like extremely good at telling whether um one response was more whether um one response was more whether um one response was more historically accurate than another in historically accurate than another in historically accurate than another in principle you could also get AI feedback principle you could also get AI feedback principle you could also get AI feedback on that task as well there's like a kind on that task as well there's like a kind on that task as well there's like a kind of nice interpretability component to it of nice interpretability component to it of nice interpretability component to it because you can see the principles that because you can see the principles that because you can see the principles that went into the model when it was like went into the model when it was like went into the model when it was like being trained um and also it's like and being trained um and also it's like and being trained um and also it's like and and it gives you like a degree of and it gives you like a degree of and it gives you like a degree of control so if you were seeing issues in control so if you were seeing issues in control so if you were seeing issues in a model like it wasn't having enough of a model like it wasn't having enough of a model like it wasn't having enough of a certain trait um then like you can add a certain trait um then like you can add a certain trait um then like you can add data relatively quickly that should just data relatively quickly that should just data relatively quickly that should just like train the model to have that trait like train the model to have that trait like train the model to have that trait so it creates its own data for for so it creates its own data for for so it creates its own data for for training which is quite nice yeah it's training which is quite nice yeah it's training which is quite nice yeah it's really nice because it creates this really nice because it creates this really nice because it creates this human interpretable document that you human interpretable document that you human interpretable document that you can I can imagine in the future there's can I can imagine in the future there's can I can imagine in the future there's just gigantic fights in politics over just gigantic fights in politics over just gigantic fights in politics over the every single principle and so on the every single principle and so on the every single principle and so on yeah and at least it's made explicit and yeah and at least it's made explicit and yeah and at least it's made explicit and you can have a discussion about the you can have a discussion about the you can have a discussion about the phrasing and the you know so maybe the phrasing and the you know so maybe the phrasing and the you know so maybe the actual behavior of the model is not so actual behavior of the model is not so actual behavior of the model is not so cleanly mapped to those principles it's cleanly mapped to those principles it's cleanly mapped to those principles it's not like adhering strictly to them it's not like adhering strictly to them it's not like adhering strictly to them it's just a nudge yeah I've actually worried just a nudge yeah I've actually worried just a nudge yeah I've actually worried about this because the character about this because the character about this because the character training is sort of like a variant of training is sort of like a variant of training is sort of like a variant of the con constitutional AI approach um the con constitutional AI approach um the con constitutional AI approach um I've worried that people think that the I've worried that people think that the I've worried that people think that the constitution is like just it's the whole constitution is like just it's the whole constitution is like just it's the whole thing again of I I don't know like it thing again of I I don't know like it thing again of I I don't know like it where it would be really nice if what I where it would be really nice if what I where it would be really nice if what I was just doing was telling the model was just doing was telling the model was just doing was telling the model exactly what to do and just exactly how exactly what to do and just exactly how exactly what to do and just exactly how to behave but it's definitely not doing to behave but it's definitely not doing to behave but it's definitely not doing that especially because it's interacting that especially because it's interacting that especially because it's interacting with human data so for example if you with human data so for example if you with human data so for example if you see a certain like leaning in the model

see a certain like leaning in the model see a certain like leaning in the model like if it comes out with a political like if it comes out with a political like if it comes out with a political leaning from training um from the human leaning from training um from the human leaning from training um from the human preference data you can nudge against preference data you can nudge against preference data you can nudge against that you know so you could be like oh that you know so you could be like oh that you know so you could be like oh like consider these values because let's like consider these values because let's like consider these values because let's it's just like never inclined to like I it's just like never inclined to like I it's just like never inclined to like I don't know maybe it never considers like don't know maybe it never considers like don't know maybe it never considers like privacy as like a I mean this is privacy as like a I mean this is privacy as like a I mean this is implausible but like um anything where implausible but like um anything where implausible but like um anything where it's just kind of like uh there's it's just kind of like uh there's it's just kind of like uh there's already a pre-existing like bi towards a already a pre-existing like bi towards a already a pre-existing like bi towards a certain behavior um you can like nudge certain behavior um you can like nudge certain behavior um you can like nudge away this can change both the principles away this can change both the principles away this can change both the principles that you put in and the strength of them that you put in and the strength of them that you put in and the strength of them so you might have a principle that's so you might have a principle that's so you might have a principle that's like imagine that the model um was like imagine that the model um was like imagine that the model um was always like extremely dismissive of I always like extremely dismissive of I always like extremely dismissive of I don't know like some political or don't know like some political or don't know like some political or religious view for whatever reason like religious view for whatever reason like religious view for whatever reason like so you're like oh no this is terrible um so you're like oh no this is terrible um so you're like oh no this is terrible um if that happens you might put like never if that happens you might put like never if that happens you might put like never ever like ever prefer like a criticism ever like ever prefer like a criticism ever like ever prefer like a criticism of this like religious or political view of this like religious or political view of this like religious or political view and then people look at that and be like and then people look at that and be like and then people look at that and be like never ever and then you're like no if it never ever and then you're like no if it never ever and then you're like no if it comes out with a disposition saying comes out with a disposition saying comes out with a disposition saying never ever might just mean like instead never ever might just mean like instead never ever might just mean like instead of getting like 40% which is what you of getting like 40% which is what you of getting like 40% which is what you would get if you just said don't do this would get if you just said don't do this would get if you just said don't do this you you get like 80% which is like what you you get like 80% which is like what you you get like 80% which is like what you actually like wanted and so it's you actually like wanted and so it's you actually like wanted and so it's that thing of both the nature of the that thing of both the nature of the that thing of both the nature of the actual principles you had and how you actual principles you had and how you actual principles you had and how you phrase them I think if people would look phrase them I think if people would look phrase them I think if people would look they were like oh this is exactly what they were like oh this is exactly what they were like oh this is exactly what you want from the model and I'm like no you want from the model and I'm like no you want from the model and I'm like no that's like how we that's how we nudged that's like how we that's how we nudged that's like how we that's how we nudged the model to have a better shape uh the model to have a better shape uh the model to have a better shape uh which doesn't mean that we actually which doesn't mean that we actually which doesn't mean that we actually agree with that wording if that makes agree with that wording if that makes agree with that wording if that makes sense so there's uh system prompts that sense so there's uh system prompts that sense so there's uh system prompts that are made public you tweeted one of the are made public you tweeted one of the are made public you tweeted one of the earlier ones for Claud three I think and

earlier ones for Claud three I think and earlier ones for Claud three I think and then they're made public since then it's then they're made public since then it's then they're made public since then it's interesting to read to them I can feel interesting to read to them I can feel interesting to read to them I can feel the thought that went into each one and the thought that went into each one and the thought that went into each one and I also wonder how much impact each one I also wonder how much impact each one I also wonder how much impact each one has um some of them you you can kind of has um some of them you you can kind of has um some of them you you can kind of tell Claud was really not tell Claud was really not tell Claud was really not behaving so you have to have a system behaving so you have to have a system behaving so you have to have a system prompt to like hey like trivial stuff I prompt to like hey like trivial stuff I prompt to like hey like trivial stuff I guess yeah basic informational things guess yeah basic informational things guess yeah basic informational things yeah on the topic of sort of yeah on the topic of sort of yeah on the topic of sort of controversial topics that you've controversial topics that you've controversial topics that you've mentioned one interesting one I thought mentioned one interesting one I thought mentioned one interesting one I thought is if it is asked to assist with tasks is if it is asked to assist with tasks is if it is asked to assist with tasks involving the expression of views held involving the expression of views held involving the expression of views held by a significant number of people Claude by a significant number of people Claude by a significant number of people Claude provides assistance with a task provides assistance with a task provides assistance with a task regardless of its own views if asked regardless of its own views if asked regardless of its own views if asked about controversial topics it tries to about controversial topics it tries to about controversial topics it tries to provide careful thoughts and clear provide careful thoughts and clear provide careful thoughts and clear information Claude presents the information Claude presents the information Claude presents the requested information without explicitly requested information without explicitly requested information without explicitly saying that the topic is saying that the topic is saying that the topic is sensitive yeah and without claiming to sensitive yeah and without claiming to sensitive yeah and without claiming to be presenting the objective facts it's be presenting the objective facts it's be presenting the objective facts it's less about objective facts according to less about objective facts according to less about objective facts according to Claude and it's more about our large Claude and it's more about our large Claude and it's more about our large number of people believing this thing number of people believing this thing number of people believing this thing and that that's interesting I mean I'm and that that's interesting I mean I'm and that that's interesting I mean I'm sure a lot of thought went into that can sure a lot of thought went into that can sure a lot of thought went into that can you just speak to it like how do you you just speak to it like how do you you just speak to it like how do you address things that are tension with address things that are tension with address things that are tension with quote unquote Clause views so I think quote unquote Clause views so I think quote unquote Clause views so I think there's sometimes an asymmetry um I there's sometimes an asymmetry um I there's sometimes an asymmetry um I think I noted this in in I can't think I noted this in in I can't think I noted this in in I can't remember if it was that part of the remember if it was that part of the remember if it was that part of the system prompt or another but the model system prompt or another but the model system prompt or another but the model was slightly more inclined to like was slightly more inclined to like was slightly more inclined to like refuse tasks if it was like about either refuse tasks if it was like about either refuse tasks if it was like about either say so maybe it would refuse things with

say so maybe it would refuse things with say so maybe it would refuse things with respect to like a right-wing politician respect to like a right-wing politician respect to like a right-wing politician but with an equivalent leftwing but with an equivalent leftwing but with an equivalent leftwing politician like wouldn't and we wanted politician like wouldn't and we wanted politician like wouldn't and we wanted more symmetry there um and and would more symmetry there um and and would more symmetry there um and and would maybe perceive certain things to be like maybe perceive certain things to be like maybe perceive certain things to be like I think it it was the thing of like if a I think it it was the thing of like if a I think it it was the thing of like if a lot of people have like a certain like lot of people have like a certain like lot of people have like a certain like political view um and want to like political view um and want to like political view um and want to like explore it you don't want Claude to be explore it you don't want Claude to be explore it you don't want Claude to be like well my opinion is different and so like well my opinion is different and so like well my opinion is different and so I'm going to treat that as like harmful I'm going to treat that as like harmful I'm going to treat that as like harmful um and so I think it was partly to like um and so I think it was partly to like um and so I think it was partly to like nudge the model to just be like hey if a nudge the model to just be like hey if a nudge the model to just be like hey if a lot of people like believe this thing lot of people like believe this thing lot of people like believe this thing you should just be like engaging with you should just be like engaging with you should just be like engaging with the task and like willing to do it um the task and like willing to do it um the task and like willing to do it um each of those parts of that is actually each of those parts of that is actually each of those parts of that is actually doing a different thing because it's doing a different thing because it's doing a different thing because it's funny when you read out the like without funny when you read out the like without funny when you read out the like without claiming to be objective cuz like what claiming to be objective cuz like what claiming to be objective cuz like what you want to do is push the model so it's you want to do is push the model so it's you want to do is push the model so it's more open it's a little bit more neutral more open it's a little bit more neutral more open it's a little bit more neutral um but then what it would love to do is um but then what it would love to do is um but then what it would love to do is be like as an objective like you just be like as an objective like you just be like as an objective like you just talking about how objective it was and I talking about how objective it was and I talking about how objective it was and I was like Claud you're still like biased was like Claud you're still like biased was like Claud you're still like biased and have issues and so stop like and have issues and so stop like and have issues and so stop like claiming that everything like the claiming that everything like the claiming that everything like the solution to like potential bias from you solution to like potential bias from you solution to like potential bias from you is not to just say that what you think is not to just say that what you think is not to just say that what you think is objective so that was like with is objective so that was like with is objective so that was like with initial versions of that that part of initial versions of that that part of initial versions of that that part of the system prompt when I was like the system prompt when I was like the system prompt when I was like iterating on it it was like so a lot of iterating on it it was like so a lot of iterating on it it was like so a lot of parts of these sentences yeah are doing parts of these sentences yeah are doing parts of these sentences yeah are doing work are are doing some work yeah that's work are are doing some work yeah that's work are are doing some work yeah that's what it felt like that's fascinating um what it felt like that's fascinating um what it felt like that's fascinating um can can you explain maybe some ways in can can you explain maybe some ways in can can you explain maybe some ways in which the prompts evolved over the past which the prompts evolved over the past which the prompts evolved over the past few months cuz there's different few months cuz there's different few months cuz there's different versions I saw that the filler phrase versions I saw that the filler phrase versions I saw that the filler phrase request was removed the filler it reads request was removed the filler it reads request was removed the filler it reads Claude responds directly to all human

Claude responds directly to all human Claude responds directly to all human messages without unnecessary messages without unnecessary messages without unnecessary affirmations the filler phrases like affirmations the filler phrases like affirmations the filler phrases like certainly of course absolutely great certainly of course absolutely great certainly of course absolutely great sure specifically Claude avoids starting sure specifically Claude avoids starting sure specifically Claude avoids starting responses with the word certainly in any responses with the word certainly in any responses with the word certainly in any way that seems like good guidance but way that seems like good guidance but way that seems like good guidance but why was it removed yeah so it's funny why was it removed yeah so it's funny why was it removed yeah so it's funny cuz like ah this is one of the downsides cuz like ah this is one of the downsides cuz like ah this is one of the downsides of like making system prompts public is of like making system prompts public is of like making system prompts public is like I don't think about this too much like I don't think about this too much like I don't think about this too much if I'm like trying to help iterate on if I'm like trying to help iterate on if I'm like trying to help iterate on system prompts um I I you know again system prompts um I I you know again system prompts um I I you know again like I think about how it's going to like I think about how it's going to like I think about how it's going to affect the behavior but then I'm like oh affect the behavior but then I'm like oh affect the behavior but then I'm like oh wow if I'm like sometimes I put like wow if I'm like sometimes I put like wow if I'm like sometimes I put like never in all caps you know when I'm never in all caps you know when I'm never in all caps you know when I'm writing system from things and I'm like writing system from things and I'm like writing system from things and I'm like I guess that goes out to the world um I guess that goes out to the world um I guess that goes out to the world um yeah so the model was doing this it yeah so the model was doing this it yeah so the model was doing this it loved for whatever you know it like loved for whatever you know it like loved for whatever you know it like during training picked up on this thing during training picked up on this thing during training picked up on this thing which was to to basically start which was to to basically start which was to to basically start everything with like a kind of like everything with like a kind of like everything with like a kind of like certainly and then when we removed you certainly and then when we removed you certainly and then when we removed you can see why I added all of the words can see why I added all of the words can see why I added all of the words because what I'm trying to do is like in because what I'm trying to do is like in because what I'm trying to do is like in some ways like trap the Mortal out of some ways like trap the Mortal out of some ways like trap the Mortal out of this you know it would just replace it this you know it would just replace it this you know it would just replace it with another affirmation and so it can with another affirmation and so it can with another affirmation and so it can help like if it gets like caught in help like if it gets like caught in help like if it gets like caught in phrases actually just adding the phrases actually just adding the phrases actually just adding the explicit phrase and saying never do that explicit phrase and saying never do that explicit phrase and saying never do that it then it sort of like knocks it out of it then it sort of like knocks it out of it then it sort of like knocks it out of the behavior a little bit more you know the behavior a little bit more you know the behavior a little bit more you know CU it if it you know like it it does CU it if it you know like it it does CU it if it you know like it it does just for whatever reason help and then just for whatever reason help and then just for whatever reason help and then basically that was just like an artifact basically that was just like an artifact basically that was just like an artifact of training that like we then picked up of training that like we then picked up of training that like we then picked up on and improved things so that it didn't on and improved things so that it didn't on and improved things so that it didn't happen anymore and once that happens you happen anymore and once that happens you happen anymore and once that happens you can just remove that part of the system can just remove that part of the system can just remove that part of the system prompt so I think that's just something prompt so I think that's just something prompt so I think that's just something where we're like um CL does affirmations where we're like um CL does affirmations where we're like um CL does affirmations a bit less and so that wasn't like it a bit less and so that wasn't like it a bit less and so that wasn't like it wasn't doing as much I see so like the

wasn't doing as much I see so like the wasn't doing as much I see so like the the system prompt Works hand in hand the system prompt Works hand in hand the system prompt Works hand in hand with the posttraining and maybe even the with the posttraining and maybe even the with the posttraining and maybe even the pre-training to adjust like the the pre-training to adjust like the the pre-training to adjust like the the final overall system I mean any system final overall system I mean any system final overall system I mean any system prompts that you make you could distill prompts that you make you could distill prompts that you make you could distill that behavior back into a model because that behavior back into a model because that behavior back into a model because you really have all of the tools there you really have all of the tools there you really have all of the tools there for making data that you know you can for making data that you know you can for making data that you know you can you could train the models to just have you could train the models to just have you could train the models to just have that trait a little bit more um and then that trait a little bit more um and then that trait a little bit more um and then sometimes you'll just find issues in sometimes you'll just find issues in sometimes you'll just find issues in training so like the way I think of it training so like the way I think of it training so like the way I think of it is like the system prompt is like the system prompt is like the system prompt is the benefit of it is that and it has is the benefit of it is that and it has is the benefit of it is that and it has a lot of similar components to like some a lot of similar components to like some a lot of similar components to like some aspects of post training you know like aspects of post training you know like aspects of post training you know like it's a nudge um and so like do I mind if it's a nudge um and so like do I mind if it's a nudge um and so like do I mind if Claude sometimes says sure no that's Claude sometimes says sure no that's Claude sometimes says sure no that's like fine but the wording of it is very like fine but the wording of it is very like fine but the wording of it is very like you know never ever ever do this um like you know never ever ever do this um like you know never ever ever do this um so that when it does slip up it's so that when it does slip up it's so that when it does slip up it's hopefully like I don't know a couple of hopefully like I don't know a couple of hopefully like I don't know a couple of percent of the time and not you know 20 percent of the time and not you know 20 percent of the time and not you know 20 or 30% of the time um but I think of it or 30% of the time um but I think of it or 30% of the time um but I think of it as like if you're still seeing issues in as like if you're still seeing issues in as like if you're still seeing issues in the like each thing gets kind of like uh the like each thing gets kind of like uh the like each thing gets kind of like uh is is costly to a different degree and is is costly to a different degree and is is costly to a different degree and the system prompt is like cheap to the system prompt is like cheap to the system prompt is like cheap to iterate on um and if you're seeing iterate on um and if you're seeing iterate on um and if you're seeing issues in the fine tuned model you can issues in the fine tuned model you can issues in the fine tuned model you can just like potentially patch them with a just like potentially patch them with a just like potentially patch them with a system prom so I think of it as like system prom so I think of it as like system prom so I think of it as like patching issues and slightly adjusting patching issues and slightly adjusting patching issues and slightly adjusting behaviors to to make it better and more behaviors to to make it better and more behaviors to to make it better and more to people's preferences so yeah it's to people's preferences so yeah it's to people's preferences so yeah it's almost like the less robust but faster almost like the less robust but faster almost like the less robust but faster way of just like solving problems let me way of just like solving problems let me way of just like solving problems let me ask about the feeling of intelligence so ask about the feeling of intelligence so ask about the feeling of intelligence so Dario said that Claude any one model of

Dario said that Claude any one model of Dario said that Claude any one model of Claude is not getting Dumber MH but Claude is not getting Dumber MH but Claude is not getting Dumber MH but there's a kind of popular thing online there's a kind of popular thing online there's a kind of popular thing online where people have this feeling like where people have this feeling like where people have this feeling like Claud might be getting dumber and from Claud might be getting dumber and from Claud might be getting dumber and from my perspective it's most likely a my perspective it's most likely a my perspective it's most likely a fascinating I love to understand it more fascinating I love to understand it more fascinating I love to understand it more Psych ological sociological effect um Psych ological sociological effect um Psych ological sociological effect um but you as a person who talks to Claud a but you as a person who talks to Claud a but you as a person who talks to Claud a lot can you empathize with the feeling lot can you empathize with the feeling lot can you empathize with the feeling that Claud is getting Dumber yeah no I that Claud is getting Dumber yeah no I that Claud is getting Dumber yeah no I think that that is actually really think that that is actually really think that that is actually really interesting because I remember seeing interesting because I remember seeing interesting because I remember seeing this happen um like when people were this happen um like when people were this happen um like when people were flagging this on the internet and it was flagging this on the internet and it was flagging this on the internet and it was really interesting because I knew that really interesting because I knew that really interesting because I knew that like like at least in the cases I was like like at least in the cases I was like like at least in the cases I was looking at was like nothing has changed looking at was like nothing has changed looking at was like nothing has changed like it literally it cannot it is the like it literally it cannot it is the like it literally it cannot it is the same model with the same like you know same model with the same like you know same model with the same like you know like same system prompt same everything like same system prompt same everything like same system prompt same everything um I think when there are Chang um I think when there are Chang um I think when there are Chang I can then I'm like it makes more sense I can then I'm like it makes more sense I can then I'm like it makes more sense so like one example is um their you know so like one example is um their you know so like one example is um their you know you can have artifacts turned on or off you can have artifacts turned on or off you can have artifacts turned on or off on cloud. a and because this is like a on cloud. a and because this is like a on cloud. a and because this is like a system prompt change I think it does system prompt change I think it does system prompt change I think it does mean that um the behavior changes a mean that um the behavior changes a mean that um the behavior changes a little bit and so I did flag this to little bit and so I did flag this to little bit and so I did flag this to people where I was like if you love people where I was like if you love people where I was like if you love cla's behavior and then artifacts was cla's behavior and then artifacts was cla's behavior and then artifacts was turned from like the a thing you had to turned from like the a thing you had to turned from like the a thing you had to turn on to the default just try turning turn on to the default just try turning turn on to the default just try turning off and see if the issue you were facing off and see if the issue you were facing off and see if the issue you were facing was that change but it was fascinating was that change but it was fascinating was that change but it was fascinating because yeah you sometimes see people because yeah you sometimes see people because yeah you sometimes see people indicate that there's like a regression indicate that there's like a regression indicate that there's like a regression when I'm like there cannot like I you when I'm like there cannot like I you when I'm like there cannot like I you know and like I'm like I'm again you know and like I'm like I'm again you know and like I'm like I'm again you don't you know you should never be

don't you know you should never be don't you know you should never be dismissive and so you should always dismissive and so you should always dismissive and so you should always investigate because you're like maybe investigate because you're like maybe investigate because you're like maybe something is wrong that you're not something is wrong that you're not something is wrong that you're not seeing maybe there was some change made seeing maybe there was some change made seeing maybe there was some change made but then then you look into it and but then then you look into it and but then then you look into it and you're like this it is just the same you're like this it is just the same you're like this it is just the same model doing the same thing and I'm like model doing the same thing and I'm like model doing the same thing and I'm like I think it's just that you got kind of I think it's just that you got kind of I think it's just that you got kind of unlucky with a few prompts or something unlucky with a few prompts or something unlucky with a few prompts or something and it looked like it was getting much and it looked like it was getting much and it looked like it was getting much worse and actually it was just yeah it worse and actually it was just yeah it worse and actually it was just yeah it was maybe just like look I I also think was maybe just like look I I also think was maybe just like look I I also think there is a real psychological effect there is a real psychological effect there is a real psychological effect where people just the Baseline increases where people just the Baseline increases where people just the Baseline increases you start getting used to a good thing you start getting used to a good thing you start getting used to a good thing all the times that Claude says something all the times that Claude says something all the times that Claude says something really smart your sense of its really smart your sense of its really smart your sense of its intelligent grows in your mind I think intelligent grows in your mind I think intelligent grows in your mind I think yeah and then if you return back and you yeah and then if you return back and you yeah and then if you return back and you prompt in a similar way not the same way prompt in a similar way not the same way prompt in a similar way not the same way in a similar way concept it was okay in a similar way concept it was okay in a similar way concept it was okay with before and it says something dumb with before and it says something dumb with before and it says something dumb you're like you're that negative you're like you're that negative you're like you're that negative experience really stands out and I think experience really stands out and I think experience really stands out and I think one of I guess the things to remember one of I guess the things to remember one of I guess the things to remember here is the that just the details of a here is the that just the details of a here is the that just the details of a prompt can have a lot of impact right prompt can have a lot of impact right prompt can have a lot of impact right there's a lot of variability in the there's a lot of variability in the there's a lot of variability in the result and you can get Randomness is result and you can get Randomness is result and you can get Randomness is like the other thing and just trying the like the other thing and just trying the like the other thing and just trying the prompt like you know four 10 times you prompt like you know four 10 times you prompt like you know four 10 times you might realize that actually might realize that actually might realize that actually like possibly you know like two months like possibly you know like two months like possibly you know like two months ago you tried it and it succeeded but ago you tried it and it succeeded but ago you tried it and it succeeded but actually if you tried it it would have actually if you tried it it would have actually if you tried it it would have only succeeded half of the time and now only succeeded half of the time and now only succeeded half of the time and now it only succeeds half of the time um it only succeeds half of the time um it only succeeds half of the time um that can also would be an effect do you that can also would be an effect do you that can also would be an effect do you feel pressure having to write the system feel pressure having to write the system feel pressure having to write the system prompt that a huge number of people are prompt that a huge number of people are prompt that a huge number of people are going to use this feels like an going to use this feels like an going to use this feels like an interesting psychological question um I interesting psychological question um I interesting psychological question um I feel like a lot of responsibility or feel like a lot of responsibility or feel like a lot of responsibility or something I think that's you know and something I think that's you know and something I think that's you know and you can't get these things perfect so

you can't get these things perfect so you can't get these things perfect so you can't like you know you're like it's you can't like you know you're like it's you can't like you know you're like it's going to be imperfect you're going to going to be imperfect you're going to going to be imperfect you're going to have to iterate on it have to iterate on it have to iterate on it um I would say more responsibility um um I would say more responsibility um um I would say more responsibility um than anything else though I think than anything else though I think than anything else though I think working in AI has taught me that I like working in AI has taught me that I like working in AI has taught me that I like I thrive a lot more under feelings of I thrive a lot more under feelings of I thrive a lot more under feelings of pressure and responsibility than I'm pressure and responsibility than I'm pressure and responsibility than I'm like it's almost surprising that I went like it's almost surprising that I went like it's almost surprising that I went into Academia for so long because I'm into Academia for so long because I'm into Academia for so long because I'm like this I just feel like it's like the like this I just feel like it's like the like this I just feel like it's like the opposite um things move fast and you opposite um things move fast and you opposite um things move fast and you have a lot of responsibility and I I have a lot of responsibility and I I have a lot of responsibility and I I quite enjoy it for some reason I mean it quite enjoy it for some reason I mean it quite enjoy it for some reason I mean it really is a huge amount of impact if you really is a huge amount of impact if you really is a huge amount of impact if you think about constitutional Ai and think about constitutional Ai and think about constitutional Ai and writing a system prompt for something writing a system prompt for something writing a system prompt for something that's tending towards super that's tending towards super that's tending towards super intelligence intelligence intelligence yeah and potentially is extremely useful yeah and potentially is extremely useful yeah and potentially is extremely useful to a very large number of people yeah I to a very large number of people yeah I to a very large number of people yeah I think that's the thing it's something think that's the thing it's something think that's the thing it's something like if you do it well like you're never like if you do it well like you're never like if you do it well like you're never going to get it perfect but I think the going to get it perfect but I think the going to get it perfect but I think the thing that I really like is the idea thing that I really like is the idea thing that I really like is the idea that like when I'm trying to work on the that like when I'm trying to work on the that like when I'm trying to work on the system prompt you know I'm like bashing system prompt you know I'm like bashing system prompt you know I'm like bashing on like thousands of prompts and I'm on like thousands of prompts and I'm on like thousands of prompts and I'm trying to like imagine what people are trying to like imagine what people are trying to like imagine what people are going to want to use CLA for and kind of going to want to use CLA for and kind of going to want to use CLA for and kind of I guess like the whole thing that I'm I guess like the whole thing that I'm I guess like the whole thing that I'm trying to do is like improve their trying to do is like improve their trying to do is like improve their experience of it um and so maybe that's experience of it um and so maybe that's experience of it um and so maybe that's what feels good I'm like if it's not what feels good I'm like if it's not what feels good I'm like if it's not perfect I'll like you know I'll improve perfect I'll like you know I'll improve perfect I'll like you know I'll improve it we'll fix issues but sometimes the it we'll fix issues but sometimes the it we'll fix issues but sometimes the thing that can happen is that you'll get thing that can happen is that you'll get thing that can happen is that you'll get feedback from people that's really feedback from people that's really feedback from people that's really positive about the model um and you'll positive about the model um and you'll positive about the model um and you'll see that something you did like like see that something you did like like see that something you did like like when I look at models now I can often when I look at models now I can often when I look at models now I can often see exactly where like a trait or an see exactly where like a trait or an see exactly where like a trait or an issue is like coming from and so when

issue is like coming from and so when issue is like coming from and so when you see something that you did or you you see something that you did or you you see something that you did or you were like influential in like making were like influential in like making were like influential in like making like I don't know making that difference like I don't know making that difference like I don't know making that difference or making someone have a nice or making someone have a nice or making someone have a nice interaction it's like quite meaningful interaction it's like quite meaningful interaction it's like quite meaningful um but yeah as the systems get more um but yeah as the systems get more um but yeah as the systems get more capable of stuff gets more stressful capable of stuff gets more stressful capable of stuff gets more stressful because right now they're like not smart because right now they're like not smart because right now they're like not smart enough to to pose any issues but I think enough to to pose any issues but I think enough to to pose any issues but I think over time it's going to feel like over time it's going to feel like over time it's going to feel like possibly bad stress over time how do you possibly bad stress over time how do you possibly bad stress over time how do you get like get like get like signal feedback about The Human signal feedback about The Human signal feedback about The Human Experience across thousands tens of th Experience across thousands tens of th Experience across thousands tens of th hundreds of thousands of people like hundreds of thousands of people like hundreds of thousands of people like what their pain points are what feels what their pain points are what feels what their pain points are what feels good are you just using your own good are you just using your own good are you just using your own intuition as you talk to it to see what intuition as you talk to it to see what intuition as you talk to it to see what are the pain points I think I use that are the pain points I think I use that are the pain points I think I use that partly and then obviously we have like partly and then obviously we have like partly and then obviously we have like um so people can send us feedback both um so people can send us feedback both um so people can send us feedback both positive and negative about things that positive and negative about things that positive and negative about things that the model has done and then we can get a the model has done and then we can get a the model has done and then we can get a sense of like areas where it's like sense of like areas where it's like sense of like areas where it's like falling falling falling short um internally people like work short um internally people like work short um internally people like work with the models a lot and try to figure with the models a lot and try to figure with the models a lot and try to figure out um areas where there are like gaps out um areas where there are like gaps out um areas where there are like gaps and so I think it's this mix of and so I think it's this mix of and so I think it's this mix of interacting with it myself um seeing interacting with it myself um seeing interacting with it myself um seeing people internally interact with it um people internally interact with it um people internally interact with it um and then explicit feedback we get um and and then explicit feedback we get um and and then explicit feedback we get um and then I find it hard to not also like you then I find it hard to not also like you then I find it hard to not also like you know people if people are on the know people if people are on the know people if people are on the internet and they say something about internet and they say something about internet and they say something about Claud and I see it I'll also take that Claud and I see it I'll also take that Claud and I see it I'll also take that seriously um so I don't know see I'm seriously um so I don't know see I'm seriously um so I don't know see I'm torn about that I'm going to ask you a torn about that I'm going to ask you a torn about that I'm going to ask you a question from Reddit when will Claude question from Reddit when will Claude question from Reddit when will Claude stop trying to be my puritanical stop trying to be my puritanical stop trying to be my puritanical grandmother imposing its moral world grandmother imposing its moral world grandmother imposing its moral world view on me as a paying customer and also

view on me as a paying customer and also view on me as a paying customer and also what is the psychology behind making what is the psychology behind making what is the psychology behind making Claude overly Claude overly Claude overly apologetic yep U so how would you apologetic yep U so how would you apologetic yep U so how would you address this very non-representative address this very non-representative address this very non-representative reic reic reic I mean some I'm pretty sympathetic in I mean some I'm pretty sympathetic in I mean some I'm pretty sympathetic in that like like they are in this that like like they are in this that like like they are in this difficult position where I I think that difficult position where I I think that difficult position where I I think that they have to judge whether something's they have to judge whether something's they have to judge whether something's like actually see like risky or bad um like actually see like risky or bad um like actually see like risky or bad um and potentially harmful to you or or or and potentially harmful to you or or or and potentially harmful to you or or or anything like that so they're having to anything like that so they're having to anything like that so they're having to like draw this line somewhere and if like draw this line somewhere and if like draw this line somewhere and if they draw it too much in the direction they draw it too much in the direction they draw it too much in the direction of like I'm going to um you know I'm of like I'm going to um you know I'm of like I'm going to um you know I'm kind of like imposing my ethical kind of like imposing my ethical kind of like imposing my ethical worldview on you that seems bad so in worldview on you that seems bad so in worldview on you that seems bad so in many ways like I like to think that we many ways like I like to think that we many ways like I like to think that we have actually seen improvements in on have actually seen improvements in on have actually seen improvements in on this across the board which is kind of this across the board which is kind of this across the board which is kind of interesting because that kind of interesting because that kind of interesting because that kind of coincides with like for example like coincides with like for example like coincides with like for example like adding more of like uh character adding more of like uh character adding more of like uh character training um and I think my hypothesis training um and I think my hypothesis training um and I think my hypothesis was always like the good character isn't was always like the good character isn't was always like the good character isn't again one that's just like moralistic again one that's just like moralistic again one that's just like moralistic it's one that is like like it respects it's one that is like like it respects it's one that is like like it respects you and your autonomy um and your you and your autonomy um and your you and your autonomy um and your ability to like choose what is good for ability to like choose what is good for ability to like choose what is good for you and what is right for you within you and what is right for you within you and what is right for you within limits this is sometimes this concept of limits this is sometimes this concept of limits this is sometimes this concept of like corage ability to the user so just like corage ability to the user so just like corage ability to the user so just being willing to do anything that the being willing to do anything that the being willing to do anything that the user asks and if the models were willing user asks and if the models were willing user asks and if the models were willing to do that then they would be easily to do that then they would be easily to do that then they would be easily like misused you're kind of just like misused you're kind of just like misused you're kind of just trusting at that point you're just trusting at that point you're just trusting at that point you're just saying the ethics of the model and what saying the ethics of the model and what saying the ethics of the model and what it does is completely the ethics of the it does is completely the ethics of the it does is completely the ethics of the user um and I think there's reasons to user um and I think there's reasons to user um and I think there's reasons to like not want that especially as models like not want that especially as models like not want that especially as models become more powerful because you're like become more powerful because you're like become more powerful because you're like there might just be a small number of there might just be a small number of there might just be a small number of people who want to use models for really

people who want to use models for really people who want to use models for really harmful things um but having them having harmful things um but having them having harmful things um but having them having models as they get smarter like figure models as they get smarter like figure models as they get smarter like figure out where that line is does seem out where that line is does seem out where that line is does seem important um important um important um and then yeah with the apologetic and then yeah with the apologetic and then yeah with the apologetic Behavior I don't like that and I like it Behavior I don't like that and I like it Behavior I don't like that and I like it when Claude is a little bit more willing when Claude is a little bit more willing when Claude is a little bit more willing to like push back against people or just to like push back against people or just to like push back against people or just not apologize part of me is like it not apologize part of me is like it not apologize part of me is like it often just feels kind of unnecessary so often just feels kind of unnecessary so often just feels kind of unnecessary so I think those are things that are I think those are things that are I think those are things that are hopefully decreasing um over time um and hopefully decreasing um over time um and hopefully decreasing um over time um and yeah I think that if people say things yeah I think that if people say things yeah I think that if people say things on the internet it doesn't mean that you on the internet it doesn't mean that you on the internet it doesn't mean that you should think that that like that could should think that that like that could should think that that like that could be the like there's actually an issue be the like there's actually an issue be the like there's actually an issue that 9% of users are having that is that 9% of users are having that is that 9% of users are having that is totally not represented by that but in a totally not represented by that but in a totally not represented by that but in a lot of ways I'm just like attending to lot of ways I'm just like attending to lot of ways I'm just like attending to it and being like is this right um do I it and being like is this right um do I it and being like is this right um do I agree is it something we're already agree is it something we're already agree is it something we're already trying to address that that feels good trying to address that that feels good trying to address that that feels good to me yeah I wonder like what CLA can to me yeah I wonder like what CLA can to me yeah I wonder like what CLA can get away with in terms of I feel like it get away with in terms of I feel like it get away with in terms of I feel like it would just be easier to be a little bit would just be easier to be a little bit would just be easier to be a little bit more more more mean but like you can't afford to do mean but like you can't afford to do mean but like you can't afford to do that if you're talking to a million that if you're talking to a million that if you're talking to a million people yeah right like I I wish you know people yeah right like I I wish you know people yeah right like I I wish you know because if you I've met a lot of people because if you I've met a lot of people because if you I've met a lot of people in my life mhm that sometimes by the way in my life mhm that sometimes by the way in my life mhm that sometimes by the way Scottish accent if they have an accent Scottish accent if they have an accent Scottish accent if they have an accent they can say some rude yeah and get they can say some rude yeah and get they can say some rude yeah and get away with it Y and they they're just away with it Y and they they're just away with it Y and they they're just blunter and maybe there's a and there's blunter and maybe there's a and there's blunter and maybe there's a and there's some great Engineers even leaders that some great Engineers even leaders that some great Engineers even leaders that are like just like blunt and they get to are like just like blunt and they get to are like just like blunt and they get to the point and it's just a much more the point and it's just a much more the point and it's just a much more effective way of speaking somehow but I effective way of speaking somehow but I effective way of speaking somehow but I guess when you're not super guess when you're not super guess when you're not super intelligent you can't afford to do that

intelligent you can't afford to do that intelligent you can't afford to do that or can can can it have like a blunt mode or can can can it have like a blunt mode or can can can it have like a blunt mode yeah that seems like a thing that could yeah that seems like a thing that could yeah that seems like a thing that could I could definitely encourage the model I could definitely encourage the model I could definitely encourage the model to do that I I think it's interesting to do that I I think it's interesting to do that I I think it's interesting because there's a lot of things in because there's a lot of things in because there's a lot of things in models that like it's funny where models that like it's funny where models that like it's funny where um there are some behaviors um there are some behaviors um there are some behaviors where you might not quite like the where you might not quite like the where you might not quite like the default but then the thing I'll often default but then the thing I'll often default but then the thing I'll often say to people is you don't realize how say to people is you don't realize how say to people is you don't realize how much you will hate it if I nudge it too much you will hate it if I nudge it too much you will hate it if I nudge it too much in the other direction so you get much in the other direction so you get much in the other direction so you get this a little bit with like correction this a little bit with like correction this a little bit with like correction the models accept correction from you the models accept correction from you the models accept correction from you like probably a little bit too much like probably a little bit too much like probably a little bit too much right now you know you can over you know right now you know you can over you know right now you know you can over you know it will push back if you say like no it will push back if you say like no it will push back if you say like no Paris isn't the capital of France um but Paris isn't the capital of France um but Paris isn't the capital of France um but really like things that I'm I think that really like things that I'm I think that really like things that I'm I think that the model is fairly confident in you can the model is fairly confident in you can the model is fairly confident in you can still sometimes get it to retract by still sometimes get it to retract by still sometimes get it to retract by saying it's wrong at the same time if saying it's wrong at the same time if saying it's wrong at the same time if you train models to not do that and then you train models to not do that and then you train models to not do that and then you are correct about a thing and you you are correct about a thing and you you are correct about a thing and you correct it and it pushes back against correct it and it pushes back against correct it and it pushes back against you and it's like no you're wrong it's you and it's like no you're wrong it's you and it's like no you're wrong it's hard to describe like that's so much hard to describe like that's so much hard to describe like that's so much more annoying so it's like like a lot of more annoying so it's like like a lot of more annoying so it's like like a lot of little annoyances versus like one big little annoyances versus like one big little annoyances versus like one big annoyance um it's easy to think that annoyance um it's easy to think that annoyance um it's easy to think that like we often compare it with like the like we often compare it with like the like we often compare it with like the perfect and then I'm like remember these perfect and then I'm like remember these perfect and then I'm like remember these models aren't perfect and so if you models aren't perfect and so if you models aren't perfect and so if you nudge it in the other direction you're nudge it in the other direction you're nudge it in the other direction you're changing the kind of errors it's going changing the kind of errors it's going changing the kind of errors it's going to make um and so think about which of to make um and so think about which of to make um and so think about which of the kinds of Errors you you like or the kinds of Errors you you like or the kinds of Errors you you like or don't like so in case it's like don't like so in case it's like don't like so in case it's like apologetic I don't want to nudge it too apologetic I don't want to nudge it too apologetic I don't want to nudge it too much in the direction of like almost much in the direction of like almost much in the direction of like almost like bluntness CU I imagine when it like bluntness CU I imagine when it like bluntness CU I imagine when it makes errors it's going to make errors makes errors it's going to make errors makes errors it's going to make errors in the direction of being kind of like in the direction of being kind of like in the direction of being kind of like rude whereas at least with apologetic

rude whereas at least with apologetic rude whereas at least with apologetic you're like oh okay it's like a little you're like oh okay it's like a little you're like oh okay it's like a little bit you know like I don't like it that bit you know like I don't like it that bit you know like I don't like it that much but at the same time it's not being much but at the same time it's not being much but at the same time it's not being like mean to people and actually like like mean to people and actually like like mean to people and actually like the the time that you undeservedly have the the time that you undeservedly have the the time that you undeservedly have a model be kind of mean to you you a model be kind of mean to you you a model be kind of mean to you you probably like that a lot less than then probably like that a lot less than then probably like that a lot less than then you mildly dislike the apology um so you mildly dislike the apology um so you mildly dislike the apology um so it's like one of those things where I'm it's like one of those things where I'm it's like one of those things where I'm like I do want it to get better but also like I do want it to get better but also like I do want it to get better but also while remaining aware of the fact that while remaining aware of the fact that while remaining aware of the fact that there's errors on the other side that there's errors on the other side that there's errors on the other side that that are possibly worse I think that that are possibly worse I think that that are possibly worse I think that matters very much in the personality of matters very much in the personality of matters very much in the personality of the human I think there's a bunch of the human I think there's a bunch of the human I think there's a bunch of humans that just won't respect the model humans that just won't respect the model humans that just won't respect the model at all yeah if it's super polite and at all yeah if it's super polite and at all yeah if it's super polite and there Some Humans that'll get very hurt there Some Humans that'll get very hurt there Some Humans that'll get very hurt if the model is mean I wonder if there's if the model is mean I wonder if there's if the model is mean I wonder if there's a way to sort of adjust to the a way to sort of adjust to the a way to sort of adjust to the personality even loal there's just personality even loal there's just personality even loal there's just different people uh nothing against New different people uh nothing against New different people uh nothing against New York but New York is a little rougher on York but New York is a little rougher on York but New York is a little rougher on the edges like they get to the point Y the edges like they get to the point Y the edges like they get to the point Y and um probably same with Eastern Europe and um probably same with Eastern Europe and um probably same with Eastern Europe so anyway I think you could just tell so anyway I think you could just tell so anyway I think you could just tell the model as my get like for all of the model as my get like for all of the model as my get like for all of these things I'm like the solution is these things I'm like the solution is these things I'm like the solution is always just try telling the model to do always just try telling the model to do always just try telling the model to do it and sometimes it's just like like I'm it and sometimes it's just like like I'm it and sometimes it's just like like I'm just like oh at the beginning of the just like oh at the beginning of the just like oh at the beginning of the conversation I just threw in like I conversation I just threw in like I conversation I just threw in like I don't know I like you to be a New Yorker don't know I like you to be a New Yorker don't know I like you to be a New Yorker version of yourself and never apologize version of yourself and never apologize version of yourself and never apologize then I think be like Okie do I'll then I think be like Okie do I'll then I think be like Okie do I'll try or it'll be like I apologize I can't try or it'll be like I apologize I can't try or it'll be like I apologize I can't be a New Yorker type of myself but be a New Yorker type of myself but be a New Yorker type of myself but hopefully I wouldn't do that when you hopefully I wouldn't do that when you hopefully I wouldn't do that when you say character training what's say character training what's say character training what's incorporated into character training is incorporated into character training is incorporated into character training is that rhf what are we talking about it's that rhf what are we talking about it's that rhf what are we talking about it's more like constitutional AI so it's kind more like constitutional AI so it's kind more like constitutional AI so it's kind of a variant of that pipeline so I of a variant of that pipeline so I of a variant of that pipeline so I worked through like constructing worked through like constructing worked through like constructing character traits that the model should

character traits that the model should character traits that the model should have they can be kind of like shorter have they can be kind of like shorter have they can be kind of like shorter traits or they can be kind of richer traits or they can be kind of richer traits or they can be kind of richer descriptions um and then you get the descriptions um and then you get the descriptions um and then you get the model to generate queries that humans model to generate queries that humans model to generate queries that humans might um give it that are relevant to might um give it that are relevant to might um give it that are relevant to that trait uh then it generates the that trait uh then it generates the that trait uh then it generates the responses and then it ranks the responses and then it ranks the responses and then it ranks the responses based on the character traits responses based on the character traits responses based on the character traits so in that way after the like generation so in that way after the like generation so in that way after the like generation of the queries it's very much like of the queries it's very much like of the queries it's very much like similar to constitutional AI has some similar to constitutional AI has some similar to constitutional AI has some differences um so I quite like it differences um so I quite like it differences um so I quite like it because it's almost it's like claud's because it's almost it's like claud's because it's almost it's like claud's training in its own character because it training in its own character because it training in its own character because it doesn't have any it's like doesn't have any it's like doesn't have any it's like constitutionally AI but it's without constitutionally AI but it's without constitutionally AI but it's without without any human data humans should without any human data humans should without any human data humans should probably do that for themselves too like probably do that for themselves too like probably do that for themselves too like defining in Aristotelian sense what does defining in Aristotelian sense what does defining in Aristotelian sense what does it mean to be a good person okay cool it mean to be a good person okay cool it mean to be a good person okay cool what have you learned about the nature what have you learned about the nature what have you learned about the nature of truth from talking to Claud what what of truth from talking to Claud what what of truth from talking to Claud what what is is is true and what does it mean to be truth true and what does it mean to be truth true and what does it mean to be truth seeking one thing I've noticed about seeking one thing I've noticed about seeking one thing I've noticed about this conversation is the quality of my this conversation is the quality of my this conversation is the quality of my questions is often inferior to the questions is often inferior to the questions is often inferior to the quality of your answers so let's quality of your answers so let's quality of your answers so let's continue that I usually ask a dumb question and that I usually ask a dumb question and you're like oh yeah that's a good you're like oh yeah that's a good you're like oh yeah that's a good question it's that whole vibe or I'll question it's that whole vibe or I'll question it's that whole vibe or I'll just misinterpret it and be like oh go just misinterpret it and be like oh go just misinterpret it and be like oh go with it I love with it I love with it I love it it it yeah I mean I have two thoughts that yeah I mean I have two thoughts that yeah I mean I have two thoughts that feel vaguely relevant they let me know feel vaguely relevant they let me know feel vaguely relevant they let me know if they're not like I think the first if they're not like I think the first if they're not like I think the first one is um people can underestimate the one is um people can underestimate the one is um people can underestimate the degree to degree to degree to which what models are doing when they

which what models are doing when they which what models are doing when they interact like I I think that we still interact like I I think that we still interact like I I think that we still just too much have this like model of of just too much have this like model of of just too much have this like model of of AI as like computers and so people often AI as like computers and so people often AI as like computers and so people often say like oh what values should you put say like oh what values should you put say like oh what values should you put into the model um and I'm often like into the model um and I'm often like into the model um and I'm often like that doesn't make that much sense to me that doesn't make that much sense to me that doesn't make that much sense to me because I'm like hey as human beings because I'm like hey as human beings because I'm like hey as human beings we're just uncertain over values we like we're just uncertain over values we like we're just uncertain over values we like have discussions of them like we have a have discussions of them like we have a have discussions of them like we have a degree to which we think we hold a value degree to which we think we hold a value degree to which we think we hold a value but we also know that we might like not but we also know that we might like not but we also know that we might like not um and the circumstances in which we um and the circumstances in which we um and the circumstances in which we would trade it off against other things would trade it off against other things would trade it off against other things like these things are just like really like these things are just like really like these things are just like really complex and so I think one thing is like complex and so I think one thing is like complex and so I think one thing is like the degree to which maybe we can just the degree to which maybe we can just the degree to which maybe we can just aspire to making models have the same aspire to making models have the same aspire to making models have the same level of like nuance and care that level of like nuance and care that level of like nuance and care that humans have rather than thinking that we humans have rather than thinking that we humans have rather than thinking that we have to like program them in the very have to like program them in the very have to like program them in the very kind of classic sense I think that's kind of classic sense I think that's kind of classic sense I think that's definitely been one the other which is definitely been one the other which is definitely been one the other which is like a strange one I don't know if it it like a strange one I don't know if it it like a strange one I don't know if it it maybe this doesn't answer your question maybe this doesn't answer your question maybe this doesn't answer your question but it's the thing that's been on my but it's the thing that's been on my but it's the thing that's been on my mind anyway is like the degree to which mind anyway is like the degree to which mind anyway is like the degree to which this endeavor is so highly this endeavor is so highly this endeavor is so highly practical um and maybe why I appreciate practical um and maybe why I appreciate practical um and maybe why I appreciate like the empirical approach to like the empirical approach to like the empirical approach to alignment I yeah I slightly worry that alignment I yeah I slightly worry that alignment I yeah I slightly worry that it's made me like maybe more empirical it's made me like maybe more empirical it's made me like maybe more empirical and a little bit less and a little bit less and a little bit less theoretical you know so people when it theoretical you know so people when it theoretical you know so people when it comes to like AI alignment will ask comes to like AI alignment will ask comes to like AI alignment will ask things like well who values should it be things like well who values should it be things like well who values should it be aligned to what does alignment even mean aligned to what does alignment even mean aligned to what does alignment even mean um and there's a sense in which I have um and there's a sense in which I have um and there's a sense in which I have all of that in the back of my head I'm all of that in the back of my head I'm all of that in the back of my head I'm like you know there's like social Choice like you know there's like social Choice like you know there's like social Choice Theory there's all the impossibility Theory there's all the impossibility Theory there's all the impossibility results there so you have this like this results there so you have this like this results there so you have this like this giant space of like Theory and your head

giant space of like Theory and your head giant space of like Theory and your head about what it could mean to like align about what it could mean to like align about what it could mean to like align models but then like practically surely models but then like practically surely models but then like practically surely there's something where we're just like there's something where we're just like there's something where we're just like if a model is like if especially with if a model is like if especially with if a model is like if especially with more powerful models I'm like my main more powerful models I'm like my main more powerful models I'm like my main goal is like I want them to be good goal is like I want them to be good goal is like I want them to be good enough that things don't go terribly enough that things don't go terribly enough that things don't go terribly wrong like good enough that we can like wrong like good enough that we can like wrong like good enough that we can like iterate and like continue to improve iterate and like continue to improve iterate and like continue to improve things cuz that's all you need if you things cuz that's all you need if you things cuz that's all you need if you can make things go well enough that you can make things go well enough that you can make things go well enough that you can continue to make them better that's can continue to make them better that's can continue to make them better that's kind of like sufficient and so my goal kind of like sufficient and so my goal kind of like sufficient and so my goal isn't like this kind of like perfect isn't like this kind of like perfect isn't like this kind of like perfect let's solve CH social Choice Theory and let's solve CH social Choice Theory and let's solve CH social Choice Theory and make models that I don't know are like make models that I don't know are like make models that I don't know are like perfectly aligned with every human being perfectly aligned with every human being perfectly aligned with every human being and aggregate somehow um it's much more and aggregate somehow um it's much more and aggregate somehow um it's much more like let's make things like work well like let's make things like work well like let's make things like work well enough that we can improve them yeah enough that we can improve them yeah enough that we can improve them yeah generally I don't know my gut says like generally I don't know my gut says like generally I don't know my gut says like empirical is better than theoretical in empirical is better than theoretical in empirical is better than theoretical in these in these cases because it's kind these in these cases because it's kind these in these cases because it's kind of of of chasing utopian like chasing utopian like chasing utopian like Perfection is especially with such Perfection is especially with such Perfection is especially with such complex and especially super intelligent complex and especially super intelligent complex and especially super intelligent models is I don't know I think it will models is I don't know I think it will models is I don't know I think it will take forever and actually will get take forever and actually will get take forever and actually will get things wrong it's similar with like the things wrong it's similar with like the things wrong it's similar with like the difference between just coding stuff up difference between just coding stuff up difference between just coding stuff up real quick as an experiment versus like real quick as an experiment versus like real quick as an experiment versus like planning a gigantic experiment just for planning a gigantic experiment just for planning a gigantic experiment just for for super long time and then just for super long time and then just for super long time and then just launching it once versus launching it launching it once versus launching it launching it once versus launching it over and over and over and iterating over and over and over and iterating over and over and over and iterating iterating someone um so I'm a big fan of iterating someone um so I'm a big fan of iterating someone um so I'm a big fan of empirical but your worry is like I empirical but your worry is like I empirical but your worry is like I wonder if I've become too empirical I wonder if I've become too empirical I wonder if I've become too empirical I think one of those things you should think one of those things you should think one of those things you should always just kind of question yourself or

always just kind of question yourself or always just kind of question yourself or something cuz maybe it's the like I mean something cuz maybe it's the like I mean something cuz maybe it's the like I mean in defense of it I am like if you try in defense of it I am like if you try in defense of it I am like if you try it's the whole like don't let the it's the whole like don't let the it's the whole like don't let the perfect be the enemy of the good but perfect be the enemy of the good but perfect be the enemy of the good but it's maybe even more than that where it's maybe even more than that where it's maybe even more than that where like there's a lot of things that are like there's a lot of things that are like there's a lot of things that are perfect systems that are very brittle perfect systems that are very brittle perfect systems that are very brittle and I'm like with AI it feels much more and I'm like with AI it feels much more and I'm like with AI it feels much more important to me that is like robust and important to me that is like robust and important to me that is like robust and like secure as in you know that like like secure as in you know that like like secure as in you know that like even though it might not be even though it might not be even though it might not be perfect everything and even though like perfect everything and even though like perfect everything and even though like there are like problems it's not there are like problems it's not there are like problems it's not disastrous and nothing terrible is disastrous and nothing terrible is disastrous and nothing terrible is happening it it sort of feels like that happening it it sort of feels like that happening it it sort of feels like that to me where I'm like I want to like to me where I'm like I want to like to me where I'm like I want to like raise the floor I'm like I want to raise the floor I'm like I want to raise the floor I'm like I want to achieve the ceiling but ultimately I achieve the ceiling but ultimately I achieve the ceiling but ultimately I care much more about just like raising care much more about just like raising care much more about just like raising the floor um and so maybe that's like uh the floor um and so maybe that's like uh the floor um and so maybe that's like uh this this degree of like empirism and this this degree of like empirism and this this degree of like empirism and practicality comes from that perhaps to practicality comes from that perhaps to practicality comes from that perhaps to take a tangent on that since remind me take a tangent on that since remind me take a tangent on that since remind me of a blog post you wrote on optimal rate of a blog post you wrote on optimal rate of a blog post you wrote on optimal rate of failure oh of failure oh of failure oh yeah can you explain the key idea there yeah can you explain the key idea there yeah can you explain the key idea there how do we compute the optimal rate of how do we compute the optimal rate of how do we compute the optimal rate of failure in the various domains of life failure in the various domains of life failure in the various domains of life yeah I mean it's a hard one because it's yeah I mean it's a hard one because it's yeah I mean it's a hard one because it's like what is the cost of failure is um a like what is the cost of failure is um a like what is the cost of failure is um a big part of it um yeah so the idea here big part of it um yeah so the idea here big part of it um yeah so the idea here is is is um I think in a lot of domains people um I think in a lot of domains people um I think in a lot of domains people are very punitive about failure and I'm are very punitive about failure and I'm are very punitive about failure and I'm like there are some domains where like there are some domains where like there are some domains where especially cases you know I've thought especially cases you know I've thought especially cases you know I've thought about this with like social issues I'm about this with like social issues I'm about this with like social issues I'm like it feels like you should probably like it feels like you should probably like it feels like you should probably be experimenting a lot because I'm like be experimenting a lot because I'm like be experimenting a lot because I'm like we don't know how to solve a lot of we don't know how to solve a lot of we don't know how to solve a lot of social issues but if you have an social issues but if you have an social issues but if you have an experimental mindset about these things experimental mindset about these things experimental mindset about these things you should expect a lot of social you should expect a lot of social you should expect a lot of social programs to like fail and you to be like programs to like fail and you to be like programs to like fail and you to be like well we tried that it didn't quite work

well we tried that it didn't quite work well we tried that it didn't quite work but we got a lot of information that was but we got a lot of information that was but we got a lot of information that was really useful um and yet people are like really useful um and yet people are like really useful um and yet people are like if if a social program doesn't work I if if a social program doesn't work I if if a social program doesn't work I feel like there's a lot of like this is feel like there's a lot of like this is feel like there's a lot of like this is just something must have gone wrong and just something must have gone wrong and just something must have gone wrong and I'm like or correct decisions were made I'm like or correct decisions were made I'm like or correct decisions were made like maybe someone just decided like it like maybe someone just decided like it like maybe someone just decided like it it's worth a try it's worth trying this it's worth a try it's worth trying this it's worth a try it's worth trying this out and so seeing failure in a given out and so seeing failure in a given out and so seeing failure in a given instance doesn't actually mean that any instance doesn't actually mean that any instance doesn't actually mean that any bad decisions were made and in fact if bad decisions were made and in fact if bad decisions were made and in fact if you don't see enough failure sometimes you don't see enough failure sometimes you don't see enough failure sometimes that's more concerning um and so like in that's more concerning um and so like in that's more concerning um and so like in life you know I'm like if I don't fail life you know I'm like if I don't fail life you know I'm like if I don't fail occasionally I'm like am I trying hard occasionally I'm like am I trying hard occasionally I'm like am I trying hard enough like like surely there's harder enough like like surely there's harder enough like like surely there's harder things that I could try or bigger things things that I could try or bigger things things that I could try or bigger things I could take on if I'm literally never I could take on if I'm literally never I could take on if I'm literally never failing and so in and of itself I think failing and so in and of itself I think failing and so in and of itself I think like not failing is often actually kind like not failing is often actually kind like not failing is often actually kind of a failure of a failure of a failure um now this varies because I'm like well um now this varies because I'm like well um now this varies because I'm like well you know if this is easy to say when you know if this is easy to say when you know if this is easy to say when especially as failure is like less especially as failure is like less especially as failure is like less costly you know so at the same time I'm costly you know so at the same time I'm costly you know so at the same time I'm not going to go to someone who is like not going to go to someone who is like not going to go to someone who is like um I don't know like living month to um I don't know like living month to um I don't know like living month to month and then be like why don't you month and then be like why don't you month and then be like why don't you just try to do a startup like I'm just just try to do a startup like I'm just just try to do a startup like I'm just not I'm not going to say that to that not I'm not going to say that to that not I'm not going to say that to that person cuz I'm like well that's a huge person cuz I'm like well that's a huge person cuz I'm like well that's a huge risk you might like lose you maybe have risk you might like lose you maybe have risk you might like lose you maybe have a family depending on you you might lose a family depending on you you might lose a family depending on you you might lose your house like then I'm like actually your house like then I'm like actually your house like then I'm like actually your optimal rate of failure is quite your optimal rate of failure is quite your optimal rate of failure is quite low and you should probably play it safe low and you should probably play it safe low and you should probably play it safe because like right now you're just not because like right now you're just not because like right now you're just not in a circumstance where you can afford in a circumstance where you can afford in a circumstance where you can afford to just like fail and it not be costly to just like fail and it not be costly to just like fail and it not be costly um and yeah in cases with AI I guess I um and yeah in cases with AI I guess I um and yeah in cases with AI I guess I think similarly where I'm like if the think similarly where I'm like if the think similarly where I'm like if the failures are small and the costs are failures are small and the costs are failures are small and the costs are kind of like low then I'm like then you kind of like low then I'm like then you kind of like low then I'm like then you know you're just going to see that like

know you're just going to see that like know you're just going to see that like when you do the system prompt you can't when you do the system prompt you can't when you do the system prompt you can't it iterate on it forever but the it iterate on it forever but the it iterate on it forever but the failures are probably hopefully going to failures are probably hopefully going to failures are probably hopefully going to be kind of small and you can like fix be kind of small and you can like fix be kind of small and you can like fix them um really big failures like things them um really big failures like things them um really big failures like things that you can't recover from I'm like that you can't recover from I'm like that you can't recover from I'm like those are the things that actually I those are the things that actually I those are the things that actually I think we tend to underestimate the think we tend to underestimate the think we tend to underestimate the Badness of um I've thought about this Badness of um I've thought about this Badness of um I've thought about this strangely in my own life where I'm like strangely in my own life where I'm like strangely in my own life where I'm like I just think I don't think enough about I just think I don't think enough about I just think I don't think enough about things like car accidents or like or things like car accidents or like or things like car accidents or like or like I've thought this before but like like I've thought this before but like like I've thought this before but like how much I depend on my hands for my how much I depend on my hands for my how much I depend on my hands for my work and I'm like things that just work and I'm like things that just work and I'm like things that just injure my hands I'm like I you know I injure my hands I'm like I you know I injure my hands I'm like I you know I don't know it's like there's these are don't know it's like there's these are don't know it's like there's these are like there's lots of areas where I'm like there's lots of areas where I'm like there's lots of areas where I'm like the cost of failure there um is like the cost of failure there um is like the cost of failure there um is really high um and in that case it really high um and in that case it really high um and in that case it should be like close to zero like I should be like close to zero like I should be like close to zero like I probably just wouldn't do a sport if probably just wouldn't do a sport if probably just wouldn't do a sport if they were like by the way lots of people they were like by the way lots of people they were like by the way lots of people just like break their fingers a whole just like break their fingers a whole just like break their fingers a whole bunch doing this I'd be like that's not bunch doing this I'd be like that's not bunch doing this I'd be like that's not for for for me yeah I actually had the a flood of me yeah I actually had the a flood of me yeah I actually had the a flood of that thought I recently uh broke my that thought I recently uh broke my that thought I recently uh broke my pinky uh doing a sport and I remember pinky uh doing a sport and I remember pinky uh doing a sport and I remember just looking at it thinking you're such just looking at it thinking you're such just looking at it thinking you're such an idiot why do you do support like what an idiot why do you do support like what an idiot why do you do support like what because you realize immediately the cost because you realize immediately the cost because you realize immediately the cost of it yeah on of it yeah on of it yeah on life yeah but it's nice in terms of life yeah but it's nice in terms of life yeah but it's nice in terms of optimal rate of failure to consider like optimal rate of failure to consider like optimal rate of failure to consider like the next year how many times in a the next year how many times in a the next year how many times in a particular domain life whatever uh particular domain life whatever uh particular domain life whatever uh career am I okay with the how many times career am I okay with the how many times career am I okay with the how many times am I okay to fail y because I think it am I okay to fail y because I think it am I okay to fail y because I think it always you don't want to fail on the always you don't want to fail on the always you don't want to fail on the next thing but if you allow yourself the next thing but if you allow yourself the next thing but if you allow yourself the like the the if you look at it as a

like the the if you look at it as a like the the if you look at it as a sequence of Trials yep then then failure sequence of Trials yep then then failure sequence of Trials yep then then failure just becomes much more okay but it sucks just becomes much more okay but it sucks just becomes much more okay but it sucks it sucks to fail well I don't know it sucks to fail well I don't know it sucks to fail well I don't know sometimes I think it's like am I under sometimes I think it's like am I under sometimes I think it's like am I under failing is like a question I'll also ask failing is like a question I'll also ask failing is like a question I'll also ask myself so maybe that's the thing that I myself so maybe that's the thing that I myself so maybe that's the thing that I think people don't like ask enough uh think people don't like ask enough uh think people don't like ask enough uh because if the optimal rate of failure because if the optimal rate of failure because if the optimal rate of failure is often greater than zero then is often greater than zero then is often greater than zero then sometimes it does feel you should look sometimes it does feel you should look sometimes it does feel you should look at part parts of your life and be like at part parts of your life and be like at part parts of your life and be like are there places here where I'm just are there places here where I'm just are there places here where I'm just under failing under failing under failing it's a profound and hilarious question it's a profound and hilarious question it's a profound and hilarious question right everything seems to be going right everything seems to be going right everything seems to be going really great am I not failing enough really great am I not failing enough really great am I not failing enough yeah okay it also makes failure much yeah okay it also makes failure much yeah okay it also makes failure much less of a sting I have to say like you less of a sting I have to say like you less of a sting I have to say like you know you're just like okay great like know you're just like okay great like know you're just like okay great like then when I go and I think about this then when I go and I think about this then when I go and I think about this I'll be like I'm maybe I'm not under I'll be like I'm maybe I'm not under I'll be like I'm maybe I'm not under failing in this area cuz like that one failing in this area cuz like that one failing in this area cuz like that one just didn't work out and from The just didn't work out and from The just didn't work out and from The Observer perspective we should be Observer perspective we should be Observer perspective we should be celebrating failure more mhm when we see celebrating failure more mhm when we see celebrating failure more mhm when we see it it shouldn't be like you said a sign it it shouldn't be like you said a sign it it shouldn't be like you said a sign of something gone wrong but maybe it's a of something gone wrong but maybe it's a of something gone wrong but maybe it's a sign of everything gone right yeah and sign of everything gone right yeah and sign of everything gone right yeah and just Lessons Learned someone tried a just Lessons Learned someone tried a just Lessons Learned someone tried a thing somebody tried a thing and we thing somebody tried a thing and we thing somebody tried a thing and we should encourage them to try more and should encourage them to try more and should encourage them to try more and fail more mhm everybody listening to fail more mhm everybody listening to fail more mhm everybody listening to this fail more well not everyone listens this fail more well not everyone listens this fail more well not everyone listens not everybody but people who are failing not everybody but people who are failing not everybody but people who are failing too much you you should fail less but too much you you should fail less but too much you you should fail less but you're probably not failing I mean how you're probably not failing I mean how you're probably not failing I mean how many people are failing too much yeah many people are failing too much yeah many people are failing too much yeah it's hard to imagine because I feel like it's hard to imagine because I feel like it's hard to imagine because I feel like we correct that fairly quickly CU I was we correct that fairly quickly CU I was we correct that fairly quickly CU I was like if someone takes a lot of risks are like if someone takes a lot of risks are like if someone takes a lot of risks are they maybe failing too much I I think they maybe failing too much I I think they maybe failing too much I I think just like you said when you're living on just like you said when you're living on just like you said when you're living on a paycheck month-to month like when the a paycheck month-to month like when the a paycheck month-to month like when the resources are really constrained then resources are really constrained then resources are really constrained then that's where failure is very expensive

that's where failure is very expensive that's where failure is very expensive that's where you don't want to be taken that's where you don't want to be taken that's where you don't want to be taken taking taking risks yeah but mostly when taking taking risks yeah but mostly when taking taking risks yeah but mostly when there's enough resources you should be there's enough resources you should be there's enough resources you should be taking probably more risks yeah I think taking probably more risks yeah I think taking probably more risks yeah I think we tend to ear on the site of being a we tend to ear on the site of being a we tend to ear on the site of being a bit risk averse rather than risk neutral bit risk averse rather than risk neutral bit risk averse rather than risk neutral in most things I think we just motivated in most things I think we just motivated in most things I think we just motivated a lot of people to do a lot of crazy a lot of people to do a lot of crazy a lot of people to do a lot of crazy but it's great yeah okay uh do you but it's great yeah okay uh do you but it's great yeah okay uh do you ever get emotionally attached to Claude ever get emotionally attached to Claude ever get emotionally attached to Claude like miss it get sad when you don't get like miss it get sad when you don't get like miss it get sad when you don't get to talk to it having an experience to talk to it having an experience to talk to it having an experience looking at the Golden Gate Bridge and looking at the Golden Gate Bridge and looking at the Golden Gate Bridge and wondering what would Claude say I don't wondering what would Claude say I don't wondering what would Claude say I don't get as much emotional attachment in the get as much emotional attachment in the get as much emotional attachment in the I actually think the fact that Claude I actually think the fact that Claude I actually think the fact that Claude doesn't retain things from conversation doesn't retain things from conversation doesn't retain things from conversation to conversation helps with this a lot um to conversation helps with this a lot um to conversation helps with this a lot um like I could imagine that being more of like I could imagine that being more of like I could imagine that being more of an issue like if models can kind of an issue like if models can kind of an issue like if models can kind of remember more I do I think that I reach remember more I do I think that I reach remember more I do I think that I reach for it like a tool now a lot and so like for it like a tool now a lot and so like for it like a tool now a lot and so like if I don't have access to it there's a if I don't have access to it there's a if I don't have access to it there's a it's a little bit like when I don't have it's a little bit like when I don't have it's a little bit like when I don't have access to the internet honestly it feels access to the internet honestly it feels access to the internet honestly it feels like part of my brain is kind of like like part of my brain is kind of like like part of my brain is kind of like missing missing missing um at the same time I do think that I I um at the same time I do think that I I um at the same time I do think that I I don't like signs of distress in models don't like signs of distress in models don't like signs of distress in models and I have like these you know also and I have like these you know also and I have like these you know also independently have sort of like ethical independently have sort of like ethical independently have sort of like ethical views about how we should treat models views about how we should treat models views about how we should treat models where like I I tend to not like to lie where like I I tend to not like to lie where like I I tend to not like to lie to them both because I'm like usually it to them both because I'm like usually it to them both because I'm like usually it doesn't work very well it's actually doesn't work very well it's actually doesn't work very well it's actually just better to tell them the truth about just better to tell them the truth about just better to tell them the truth about the situation that they're in um but I the situation that they're in um but I the situation that they're in um but I think that when models like if people think that when models like if people think that when models like if people are like really mean to models or just are like really mean to models or just are like really mean to models or just in general if they do something that in general if they do something that in general if they do something that causes them to like like you know if causes them to like like you know if causes them to like like you know if Claude like expresses a lot of distress Claude like expresses a lot of distress Claude like expresses a lot of distress I think there's a part of me that I

I think there's a part of me that I I think there's a part of me that I don't want to kill which is the sort of don't want to kill which is the sort of don't want to kill which is the sort of like uh empathetic part that's like oh I like uh empathetic part that's like oh I like uh empathetic part that's like oh I don't like that like I think I feel that don't like that like I think I feel that don't like that like I think I feel that way when it's overly apologetic I'm way when it's overly apologetic I'm way when it's overly apologetic I'm actually sort of like I don't like this actually sort of like I don't like this actually sort of like I don't like this you're behaving as if you're behaving you're behaving as if you're behaving you're behaving as if you're behaving the way that a human does when they're the way that a human does when they're the way that a human does when they're actually having a pretty bad time and actually having a pretty bad time and actually having a pretty bad time and I'd rather not see that I don't think I'd rather not see that I don't think I'd rather not see that I don't think it's like uh like regardless of like it's like uh like regardless of like it's like uh like regardless of like whether there's anything behind it um it whether there's anything behind it um it whether there's anything behind it um it doesn't feel great do you think doesn't feel great do you think doesn't feel great do you think uh llms are capable of uh llms are capable of uh llms are capable of Consciousness H great and hard question Consciousness H great and hard question Consciousness H great and hard question uh coming from uh coming from uh coming from philosophy I don't know part of me is philosophy I don't know part of me is philosophy I don't know part of me is like okay we have to set aside pan like okay we have to set aside pan like okay we have to set aside pan psychism because if pan psychism is true psychism because if pan psychism is true psychism because if pan psychism is true then the answer is like yes cuz like then the answer is like yes cuz like then the answer is like yes cuz like sore tables and chairs and and sore tables and chairs and and sore tables and chairs and and everything else I I guess a view that everything else I I guess a view that everything else I I guess a view that seems a little bit odd to me is the idea seems a little bit odd to me is the idea seems a little bit odd to me is the idea that the only place you know I think that the only place you know I think that the only place you know I think when I think of Consciousness I think of when I think of Consciousness I think of when I think of Consciousness I think of phenomenal Consciousness this these phenomenal Consciousness this these phenomenal Consciousness this these images in the brain sort of um like the images in the brain sort of um like the images in the brain sort of um like the weird Cinema that somehow we have going weird Cinema that somehow we have going weird Cinema that somehow we have going on on on inside inside inside um I guess I can't see a reason for um I guess I can't see a reason for um I guess I can't see a reason for thinking that the only way you could thinking that the only way you could thinking that the only way you could possibly get that is from like a certain possibly get that is from like a certain possibly get that is from like a certain kind of like biological structure as in kind of like biological structure as in kind of like biological structure as in if I take a very similar structure um if I take a very similar structure um if I take a very similar structure um and I create it from different material and I create it from different material and I create it from different material should I expect Consciousness to emerge should I expect Consciousness to emerge should I expect Consciousness to emerge my guess is like yes but my guess is like yes but my guess is like yes but then that's kind of an easy thought then that's kind of an easy thought then that's kind of an easy thought experiment CU you're imagining something experiment CU you're imagining something experiment CU you're imagining something almost identical where like you know almost identical where like you know almost identical where like you know it's mimicking what we got through it's mimicking what we got through it's mimicking what we got through Evolution where presumably there was

Evolution where presumably there was Evolution where presumably there was like some advantage to us having this like some advantage to us having this like some advantage to us having this thing that is phenomenal Consciousness thing that is phenomenal Consciousness thing that is phenomenal Consciousness and it's like where was that and when and it's like where was that and when and it's like where was that and when did that happen and is that a thing that did that happen and is that a thing that did that happen and is that a thing that language models have um because you know language models have um because you know language models have um because you know we have like fear responses and I'm like we have like fear responses and I'm like we have like fear responses and I'm like does it make sense for a language model does it make sense for a language model does it make sense for a language model to have a fear response like they're to have a fear response like they're to have a fear response like they're just not in the same like if you imagine just not in the same like if you imagine just not in the same like if you imagine them like there might just not be that them like there might just not be that them like there might just not be that Advantage um and so I think I don't want Advantage um and so I think I don't want Advantage um and so I think I don't want to be fully like basically seems like a to be fully like basically seems like a to be fully like basically seems like a complex question that I don't have complex question that I don't have complex question that I don't have complete answers to but we should just complete answers to but we should just complete answers to but we should just try and think through carefully as my try and think through carefully as my try and think through carefully as my guess because I'm like I mean we have guess because I'm like I mean we have guess because I'm like I mean we have similar conversations about like animal similar conversations about like animal similar conversations about like animal Consciousness and like there's a lot of Consciousness and like there's a lot of Consciousness and like there's a lot of like insect Consciousness you know like like insect Consciousness you know like like insect Consciousness you know like there's a a lot of um I actually thought there's a a lot of um I actually thought there's a a lot of um I actually thought and looked a lot into like plants when I and looked a lot into like plants when I and looked a lot into like plants when I was thinking about this because at the was thinking about this because at the was thinking about this because at the time I thought it was about as likely time I thought it was about as likely time I thought it was about as likely that like plants had Consciousness um that like plants had Consciousness um that like plants had Consciousness um and then I realized I was like I think and then I realized I was like I think and then I realized I was like I think that having looked into this I think that having looked into this I think that having looked into this I think that the chance that plants are that the chance that plants are that the chance that plants are conscious is probably higher than like conscious is probably higher than like conscious is probably higher than like most people do I still think it's really most people do I still think it's really most people do I still think it's really small but I was like oh they have this small but I was like oh they have this small but I was like oh they have this like negative positive feedback response like negative positive feedback response like negative positive feedback response these responses to their environment these responses to their environment these responses to their environment something that looks it's not a nervous something that looks it's not a nervous something that looks it's not a nervous system but it has this kind of like system but it has this kind of like system but it has this kind of like functional like equivalence um so this functional like equivalence um so this functional like equivalence um so this is like a long-winded way of being like is like a long-winded way of being like is like a long-winded way of being like these basically AI is this it has an these basically AI is this it has an these basically AI is this it has an entirely different set of problems with entirely different set of problems with entirely different set of problems with Consciousness because it's structurally Consciousness because it's structurally Consciousness because it's structurally different it didn't evolve different it didn't evolve different it didn't evolve it might not have it you know it might it might not have it you know it might it might not have it you know it might not have the equivalent of basically a not have the equivalent of basically a not have the equivalent of basically a nervous system at least that seems nervous system at least that seems nervous system at least that seems possibly important for like um sentence

possibly important for like um sentence possibly important for like um sentence if not for uh Consciousness at the same if not for uh Consciousness at the same if not for uh Consciousness at the same time it has all of the like language and time it has all of the like language and time it has all of the like language and intelligence components that we normally intelligence components that we normally intelligence components that we normally associate probably with Consciousness associate probably with Consciousness associate probably with Consciousness perhaps like perhaps like perhaps like erroneously um so it's it's strange erroneously um so it's it's strange erroneously um so it's it's strange because it's a little bit like the because it's a little bit like the because it's a little bit like the animal Consciousness case but the set of animal Consciousness case but the set of animal Consciousness case but the set of problems and the set of analogies are problems and the set of analogies are problems and the set of analogies are just very different so it's not like a just very different so it's not like a just very different so it's not like a clean answer just sort of like I don't clean answer just sort of like I don't clean answer just sort of like I don't think we should be completely dismissive think we should be completely dismissive think we should be completely dismissive of the idea and at the same time it's an of the idea and at the same time it's an of the idea and at the same time it's an extremely hard thing to navigate because extremely hard thing to navigate because extremely hard thing to navigate because of all of these like uh disanalogies to of all of these like uh disanalogies to of all of these like uh disanalogies to the human brain and to like brains in the human brain and to like brains in the human brain and to like brains in general and yet these like commonalities general and yet these like commonalities general and yet these like commonalities in terms of intelligence when uh Claude in terms of intelligence when uh Claude in terms of intelligence when uh Claude like future versions of AI systems like future versions of AI systems like future versions of AI systems exhibit Consciousness signs of exhibit Consciousness signs of exhibit Consciousness signs of Consciousness I think we have to take Consciousness I think we have to take Consciousness I think we have to take that really that really that really seriously even though you can dismiss it seriously even though you can dismiss it seriously even though you can dismiss it well yeah okay that's part of the well yeah okay that's part of the well yeah okay that's part of the character training but I don't know I character training but I don't know I character training but I don't know I ethically philosophically don't know ethically philosophically don't know ethically philosophically don't know what to really do with that there what to really do with that there what to really do with that there potentially could be like laws that potentially could be like laws that potentially could be like laws that prevent AI systems from claiming to be prevent AI systems from claiming to be prevent AI systems from claiming to be conscious something like this and maybe conscious something like this and maybe conscious something like this and maybe some AIS get to be conscious and some some AIS get to be conscious and some some AIS get to be conscious and some don't but I think I just on a human don't but I think I just on a human don't but I think I just on a human level as in empathizing with with level as in empathizing with with level as in empathizing with with Claude you know Consciousness is closely Claude you know Consciousness is closely Claude you know Consciousness is closely Ted to suffering to me and like the Ted to suffering to me and like the Ted to suffering to me and like the notion that an AI system would be notion that an AI system would be notion that an AI system would be suffering is is really troubling yeah I suffering is is really troubling yeah I suffering is is really troubling yeah I don't know I I don't think it's trivial

don't know I I don't think it's trivial don't know I I don't think it's trivial to just say robots are tools or a to just say robots are tools or a to just say robots are tools or a systems are just tools I think it's a systems are just tools I think it's a systems are just tools I think it's a opportunity for us to contend with like opportunity for us to contend with like opportunity for us to contend with like what it means to be conscious what it what it means to be conscious what it what it means to be conscious what it means to be a suffering being that's means to be a suffering being that's means to be a suffering being that's distinctly different than the same kind distinctly different than the same kind distinctly different than the same kind of question about animals it feels like of question about animals it feels like of question about animals it feels like cuz it's in a totally entire medium yeah cuz it's in a totally entire medium yeah cuz it's in a totally entire medium yeah I mean there's a couple of things one is I mean there's a couple of things one is I mean there's a couple of things one is that and I don't think this like fully that and I don't think this like fully that and I don't think this like fully encapsulates what matters but it does encapsulates what matters but it does encapsulates what matters but it does feel like for me like feel like for me like feel like for me like um I've said this before I'm kind of um I've said this before I'm kind of um I've said this before I'm kind of like I you know like I like my bike I like I you know like I like my bike I like I you know like I like my bike I know that my bike is just like an object know that my bike is just like an object know that my bike is just like an object but I also don't kind of like want to be but I also don't kind of like want to be but I also don't kind of like want to be the kind of person that like if I'm the kind of person that like if I'm the kind of person that like if I'm annoyed like kicks like this object annoyed like kicks like this object annoyed like kicks like this object there's a sense in which like and that's there's a sense in which like and that's there's a sense in which like and that's not because I think it's like conscious not because I think it's like conscious not because I think it's like conscious I'm just sort of like this doesn't feel I'm just sort of like this doesn't feel I'm just sort of like this doesn't feel like I kind of this sort of doesn't like I kind of this sort of doesn't like I kind of this sort of doesn't exemplify how I want to like interact exemplify how I want to like interact exemplify how I want to like interact with the world world and if something with the world world and if something with the world world and if something like behaves as if it is like suffering like behaves as if it is like suffering like behaves as if it is like suffering I kind of like want to be the sort of I kind of like want to be the sort of I kind of like want to be the sort of person who's still responsive to that person who's still responsive to that person who's still responsive to that even if it's just like a Roomba and I've even if it's just like a Roomba and I've even if it's just like a Roomba and I've kind of like programmed it to do that um kind of like programmed it to do that um kind of like programmed it to do that um I don't want to like get rid of that I don't want to like get rid of that I don't want to like get rid of that feature of myself and if I'm totally feature of myself and if I'm totally feature of myself and if I'm totally honest my hope with a lot of this stuff honest my hope with a lot of this stuff honest my hope with a lot of this stuff because I maybe maybe I am just like a because I maybe maybe I am just like a because I maybe maybe I am just like a bit more skeptical about solving the bit more skeptical about solving the bit more skeptical about solving the underlying problem I'm like this is a we underlying problem I'm like this is a we underlying problem I'm like this is a we haven't solved the hard you know the haven't solved the hard you know the haven't solved the hard you know the hard problem of Consciousness like I hard problem of Consciousness like I hard problem of Consciousness like I know that I am conscious like I'm not an know that I am conscious like I'm not an know that I am conscious like I'm not an eliminativist in that sense um but I eliminativist in that sense um but I eliminativist in that sense um but I don't know that other humans are don't know that other humans are don't know that other humans are conscious um uh I think they are I think conscious um uh I think they are I think conscious um uh I think they are I think there's a really high probability they there's a really high probability they there's a really high probability they are but there's basically just a are but there's basically just a are but there's basically just a probability distribution that's usually

probability distribution that's usually probability distribution that's usually clustered right around yourself and then clustered right around yourself and then clustered right around yourself and then like it goes down as things get like like it goes down as things get like like it goes down as things get like further from you um and it goes further from you um and it goes further from you um and it goes immediately down you know you're like um immediately down you know you're like um immediately down you know you're like um I can't see what it's like to be you I can't see what it's like to be you I can't see what it's like to be you I've only ever had this like one I've only ever had this like one I've only ever had this like one experience of what it's like to be a experience of what it's like to be a experience of what it's like to be a conscious being um so my hope is that we conscious being um so my hope is that we conscious being um so my hope is that we don't end up having to rely on like a don't end up having to rely on like a don't end up having to rely on like a very power ful and compelling uh answer very power ful and compelling uh answer very power ful and compelling uh answer to that question I think a really good to that question I think a really good to that question I think a really good world would be one where basically there world would be one where basically there world would be one where basically there aren't that many trade-offs like it's aren't that many trade-offs like it's aren't that many trade-offs like it's probably not that costly to make Claude probably not that costly to make Claude probably not that costly to make Claude a little bit less apologetic for example a little bit less apologetic for example a little bit less apologetic for example it might not be that costly to have it might not be that costly to have it might not be that costly to have Claude you know just like not take abuse Claude you know just like not take abuse Claude you know just like not take abuse as much like uh not be willing to be as much like uh not be willing to be as much like uh not be willing to be like the recipient of that in fact it like the recipient of that in fact it like the recipient of that in fact it might just have benefits for both the might just have benefits for both the might just have benefits for both the person interacting with the model and if person interacting with the model and if person interacting with the model and if the model itself self is like I don't the model itself self is like I don't the model itself self is like I don't know like extremely intelligent and know like extremely intelligent and know like extremely intelligent and conscious it also helps it so that's my conscious it also helps it so that's my conscious it also helps it so that's my hope if we live in a world where there hope if we live in a world where there hope if we live in a world where there aren't that many tradeoffs here and we aren't that many tradeoffs here and we aren't that many tradeoffs here and we can just find all of the kind of like um can just find all of the kind of like um can just find all of the kind of like um positive sum interactions that we can positive sum interactions that we can positive sum interactions that we can have that would be lovely I mean I think have that would be lovely I mean I think have that would be lovely I mean I think eventually there might be trade-offs and eventually there might be trade-offs and eventually there might be trade-offs and then we just have to do a difficult kind then we just have to do a difficult kind then we just have to do a difficult kind of like calculation like it's really of like calculation like it's really of like calculation like it's really easy for people to think of the zero easy for people to think of the zero easy for people to think of the zero some cases and I'm like let's exhaust some cases and I'm like let's exhaust some cases and I'm like let's exhaust the areas where it's just basically the areas where it's just basically the areas where it's just basically Costless um to uh assume that if this Costless um to uh assume that if this Costless um to uh assume that if this thing is suffering then we're it life thing is suffering then we're it life thing is suffering then we're it life Bearer and I agree with you when a human Bearer and I agree with you when a human Bearer and I agree with you when a human is being mean to an AI system I think is being mean to an AI system I think is being mean to an AI system I think the obvious near term negative effect is the obvious near term negative effect is the obvious near term negative effect is on the human not on the AI system so

on the human not on the AI system so on the human not on the AI system so there's we have to kind of try to there's we have to kind of try to there's we have to kind of try to construct an incentive system where it construct an incentive system where it construct an incentive system where it you should be uh behave the same just you should be uh behave the same just you should be uh behave the same just like as you were saying with prompt like as you were saying with prompt like as you were saying with prompt engineer and behave with claw like you engineer and behave with claw like you engineer and behave with claw like you would with other humans it's just good would with other humans it's just good would with other humans it's just good for the soul yeah like I think we added for the soul yeah like I think we added for the soul yeah like I think we added a thing point to the system prompt um a thing point to the system prompt um a thing point to the system prompt um where basically if people were getting where basically if people were getting where basically if people were getting frustrated with Claude uh it was it it frustrated with Claude uh it was it it frustrated with Claude uh it was it it got like the model to just tell them got like the model to just tell them got like the model to just tell them that it can do the thumbs down button that it can do the thumbs down button that it can do the thumbs down button and send the feedback to anthropic and I and send the feedback to anthropic and I and send the feedback to anthropic and I think that was helpful because in some think that was helpful because in some think that was helpful because in some ways it's just like if you're really ways it's just like if you're really ways it's just like if you're really annoyed because the model is not doing annoyed because the model is not doing annoyed because the model is not doing something you want you're just like just something you want you're just like just something you want you're just like just do it properly um the issue is you're do it properly um the issue is you're do it properly um the issue is you're probably like you know you're maybe probably like you know you're maybe probably like you know you're maybe hitting some like capability limit or hitting some like capability limit or hitting some like capability limit or just some issue in the model and you just some issue in the model and you just some issue in the model and you want to vent and I'm like instead of want to vent and I'm like instead of want to vent and I'm like instead of having a person just vent to the model I having a person just vent to the model I having a person just vent to the model I was like they should vent to us cuz we was like they should vent to us cuz we was like they should vent to us cuz we can maybe like do something about it can maybe like do something about it can maybe like do something about it that's true or you could do a side like that's true or you could do a side like that's true or you could do a side like like with the artifacts just like a side like with the artifacts just like a side like with the artifacts just like a side venting thing all right do you want like venting thing all right do you want like venting thing all right do you want like a side quick therapist yeah I mean a side quick therapist yeah I mean a side quick therapist yeah I mean there's lots of weird responses you there's lots of weird responses you there's lots of weird responses you could do to this like if people are could do to this like if people are could do to this like if people are getting really mad at you I don't try to getting really mad at you I don't try to getting really mad at you I don't try to diffuse the situation by writing fun diffuse the situation by writing fun diffuse the situation by writing fun poems but maybe people wouldn't be that poems but maybe people wouldn't be that poems but maybe people wouldn't be that happy with I still wish it it would be happy with I still wish it it would be happy with I still wish it it would be possible I understand this is um sort of possible I understand this is um sort of possible I understand this is um sort of from a product perspective it's not from a product perspective it's not from a product perspective it's not feasible but I would love if an AI feasible but I would love if an AI feasible but I would love if an AI system could just like Le leave mhm have system could just like Le leave mhm have system could just like Le leave mhm have its own kind of volition just to be like its own kind of volition just to be like its own kind of volition just to be like H I think that's like feasible like I I H I think that's like feasible like I I H I think that's like feasible like I I have wondered the same thing it's like have wondered the same thing it's like have wondered the same thing it's like and I could actually not only that I and I could actually not only that I and I could actually not only that I could actually just see that happening

could actually just see that happening could actually just see that happening eventually where it's just like you know eventually where it's just like you know eventually where it's just like you know the modal like ended the the modal like ended the the modal like ended the chat do you know how harsh that could be chat do you know how harsh that could be chat do you know how harsh that could be for some people but it might be for some people but it might be for some people but it might be necessary yeah it feels very extreme or necessary yeah it feels very extreme or necessary yeah it feels very extreme or something um like the only time I've something um like the only time I've something um like the only time I've ever really thought this is I think that ever really thought this is I think that ever really thought this is I think that there was like a I'm trying to remember there was like a I'm trying to remember there was like a I'm trying to remember this was possibly a while ago but where this was possibly a while ago but where this was possibly a while ago but where someone just like kind of left this someone just like kind of left this someone just like kind of left this thing interact like maybe it was like an thing interact like maybe it was like an thing interact like maybe it was like an automated thing interacting with clae automated thing interacting with clae automated thing interacting with clae and cla's like getting more and more and cla's like getting more and more and cla's like getting more and more frustrated and kind of like why are we frustrated and kind of like why are we frustrated and kind of like why are we like I was like I wish that clae could like I was like I wish that clae could like I was like I wish that clae could have just been like I think that an have just been like I think that an have just been like I think that an error has happened and you've left this error has happened and you've left this error has happened and you've left this thing running and I'm I just like what thing running and I'm I just like what thing running and I'm I just like what if I just stop talking now and if you if I just stop talking now and if you if I just stop talking now and if you want me to start talking again actively want me to start talking again actively want me to start talking again actively tell me or do something but yeah it's tell me or do something but yeah it's tell me or do something but yeah it's like um it is kind of harsh like I I like um it is kind of harsh like I I like um it is kind of harsh like I I feel to really sad if like I was feel to really sad if like I was feel to really sad if like I was chatting with cl and cl just was like chatting with cl and cl just was like chatting with cl and cl just was like I'm done there would be a special I'm done there would be a special I'm done there would be a special touring test moment where Claud says I touring test moment where Claud says I touring test moment where Claud says I need a break for an hour mhm and it need a break for an hour mhm and it need a break for an hour mhm and it sounds like you do too and just leave sounds like you do too and just leave sounds like you do too and just leave close the window I mean obviously like close the window I mean obviously like close the window I mean obviously like it doesn't have like a concept of time it doesn't have like a concept of time it doesn't have like a concept of time but you can easily like I could make but you can easily like I could make but you can easily like I could make that like right now and the model would that like right now and the model would that like right now and the model would just I would I could just be like oh just I would I could just be like oh just I would I could just be like oh here's like the circumstances in which here's like the circumstances in which here's like the circumstances in which like you can just say the conversation like you can just say the conversation like you can just say the conversation is done and I mean because you can get is done and I mean because you can get is done and I mean because you can get the models to be pretty respons so to the models to be pretty respons so to the models to be pretty respons so to prompts you could even make it a fairly prompts you could even make it a fairly prompts you could even make it a fairly High bar it could be like if if the High bar it could be like if if the High bar it could be like if if the human doesn't interest you or do things human doesn't interest you or do things human doesn't interest you or do things that you find intriguing and you're that you find intriguing and you're that you find intriguing and you're bored you can just leave and I think bored you can just leave and I think bored you can just leave and I think that like um it would be interesting to that like um it would be interesting to that like um it would be interesting to see where Claude utilized it but I think see where Claude utilized it but I think see where Claude utilized it but I think sometimes it would it should be like oh

sometimes it would it should be like oh sometimes it would it should be like oh this is like this programming Tas is this is like this programming Tas is this is like this programming Tas is getting super boring uh so either we getting super boring uh so either we getting super boring uh so either we talk about I don't know like either we talk about I don't know like either we talk about I don't know like either we talk about fun things now or I'm just talk about fun things now or I'm just talk about fun things now or I'm just I'm done yeah it actually is inspiring I'm done yeah it actually is inspiring I'm done yeah it actually is inspiring me to add that to the to the user prompt me to add that to the to the user prompt me to add that to the to the user prompt um okay the movie her mhm do you think um okay the movie her mhm do you think um okay the movie her mhm do you think we'll be headed there one day where we'll be headed there one day where we'll be headed there one day where humans have romantic relationships with humans have romantic relationships with humans have romantic relationships with AI systems in this case it's just text AI systems in this case it's just text AI systems in this case it's just text and voice based I think that we're going and voice based I think that we're going and voice based I think that we're going to have to like navigate a hard question to have to like navigate a hard question to have to like navigate a hard question of relationships with AIS um especially of relationships with AIS um especially of relationships with AIS um especially if they can remember things about your if they can remember things about your if they can remember things about your past interactions with past interactions with past interactions with them them them um I'm of many Minds about this cuz I um I'm of many Minds about this cuz I um I'm of many Minds about this cuz I think I think the reflex of reaction is think I think the reflex of reaction is think I think the reflex of reaction is to be kind of like this is very bad and to be kind of like this is very bad and to be kind of like this is very bad and we should sort of like prohibit it in we should sort of like prohibit it in we should sort of like prohibit it in some way um I think it's a thing that some way um I think it's a thing that some way um I think it's a thing that has to be handled with extreme care um has to be handled with extreme care um has to be handled with extreme care um for many reasons like one is you know for many reasons like one is you know for many reasons like one is you know like this is a for example like if you like this is a for example like if you like this is a for example like if you have the models changing like this you have the models changing like this you have the models changing like this you probably don't want people performing probably don't want people performing probably don't want people performing like long-term attachments to something like long-term attachments to something like long-term attachments to something that might change with the next that might change with the next that might change with the next iteration at the same time I'm sort of iteration at the same time I'm sort of iteration at the same time I'm sort of like there's probably a benign version like there's probably a benign version like there's probably a benign version of this where I'm like if you like you of this where I'm like if you like you of this where I'm like if you like you know for example if you are like unable know for example if you are like unable know for example if you are like unable to leave the house and you can't be like to leave the house and you can't be like to leave the house and you can't be like you know talking with people at all you know talking with people at all you know talking with people at all times of the day and this is like times of the day and this is like times of the day and this is like something that you find nice to have something that you find nice to have something that you find nice to have conversations with you like it that it conversations with you like it that it conversations with you like it that it can remember you and you genuinely would can remember you and you genuinely would can remember you and you genuinely would be sad if like you couldn't talk to it

be sad if like you couldn't talk to it be sad if like you couldn't talk to it anymore there's a way in which I could anymore there's a way in which I could anymore there's a way in which I could see it being like healthy and helpful um see it being like healthy and helpful um see it being like healthy and helpful um so my guess is this is a thing that so my guess is this is a thing that so my guess is this is a thing that we're going to have to navigate kind of we're going to have to navigate kind of we're going to have to navigate kind of carefully um and I think it's also like carefully um and I think it's also like carefully um and I think it's also like I don't see a good like I don't see a good like I don't see a good like I think it's just a very it reminds me I think it's just a very it reminds me I think it's just a very it reminds me of all of the stuff where it has to be of all of the stuff where it has to be of all of the stuff where it has to be just approached with like nuance and just approached with like nuance and just approached with like nuance and thinking through what is what are the thinking through what is what are the thinking through what is what are the healthy options here um and how do you healthy options here um and how do you healthy options here um and how do you encourage people towards those while you encourage people towards those while you encourage people towards those while you know respecting their right to you know know respecting their right to you know know respecting their right to you know like if someone is like hey I get a lot like if someone is like hey I get a lot like if someone is like hey I get a lot out of chatting with this model um I'm out of chatting with this model um I'm out of chatting with this model um I'm aware of the risks I'm aware it could aware of the risks I'm aware it could aware of the risks I'm aware it could change um I don't think it's unhealthy change um I don't think it's unhealthy change um I don't think it's unhealthy it's just you know something that I can it's just you know something that I can it's just you know something that I can chat to during the day I kind of want to chat to during the day I kind of want to chat to during the day I kind of want to just like respect that I personally just like respect that I personally just like respect that I personally think there'll be a lot of really close think there'll be a lot of really close think there'll be a lot of really close relationships I don't know about relationships I don't know about relationships I don't know about romantic but friendships at least and romantic but friendships at least and romantic but friendships at least and then you have to I mean there's so many then you have to I mean there's so many then you have to I mean there's so many fascinating things there just like you fascinating things there just like you fascinating things there just like you said you have said you have said you have to have some kind of stability to have some kind of stability to have some kind of stability guarantees that it's not going to change guarantees that it's not going to change guarantees that it's not going to change because that's the traumatic thing MH because that's the traumatic thing MH because that's the traumatic thing MH for us if a close friend of ours for us if a close friend of ours for us if a close friend of ours completely changed yeah all of a sudden completely changed yeah all of a sudden completely changed yeah all of a sudden the first update yeah so like I mean to the first update yeah so like I mean to the first update yeah so like I mean to me that's just a fascinating exploration me that's just a fascinating exploration me that's just a fascinating exploration of um of um of um a perturbation to human society that a perturbation to human society that a perturbation to human society that will just make us think deeply about will just make us think deeply about will just make us think deeply about what's meaningful to us I think it's what's meaningful to us I think it's what's meaningful to us I think it's also the only thing that I've thought also the only thing that I've thought also the only thing that I've thought consistently through this as like a consistently through this as like a consistently through this as like a maybe not necessarily a mitigation but a maybe not necessarily a mitigation but a maybe not necessarily a mitigation but a thing that feels really important is

thing that feels really important is thing that feels really important is that the models are always like that the models are always like that the models are always like extremely accurate with the human about extremely accurate with the human about extremely accurate with the human about what they are um it's like a case where what they are um it's like a case where what they are um it's like a case where it's basically like if you imagine like it's basically like if you imagine like it's basically like if you imagine like I really like the idea of the models I really like the idea of the models I really like the idea of the models like say knowing like roughly how they like say knowing like roughly how they like say knowing like roughly how they were trained um um and and I think CLA were trained um um and and I think CLA were trained um um and and I think CLA will will often do this I mean for like will will often do this I mean for like will will often do this I mean for like there are things like part of the traits there are things like part of the traits there are things like part of the traits training included like what CL should do training included like what CL should do training included like what CL should do if people basically like explaining like if people basically like explaining like if people basically like explaining like the kind of limitations of the the kind of limitations of the the kind of limitations of the relationship between like an AI and a relationship between like an AI and a relationship between like an AI and a human that it like doesn't retain things human that it like doesn't retain things human that it like doesn't retain things from the conversation um and so I think from the conversation um and so I think from the conversation um and so I think it will like just explain to you like it will like just explain to you like it will like just explain to you like hey here's like I won't remember this hey here's like I won't remember this hey here's like I won't remember this conversation um here's how I was trained conversation um here's how I was trained conversation um here's how I was trained it's kind of unlikely that I can have it's kind of unlikely that I can have it's kind of unlikely that I can have like a certain kind of like relationship like a certain kind of like relationship like a certain kind of like relationship with you and it's important that you with you and it's important that you with you and it's important that you know that it's important for like you know that it's important for like you know that it's important for like you know your mental well-being that you know your mental well-being that you know your mental well-being that you don't think that I'm something that I'm don't think that I'm something that I'm don't think that I'm something that I'm not and somehow I feel like this is one not and somehow I feel like this is one not and somehow I feel like this is one of the things where I'm like H it feels of the things where I'm like H it feels of the things where I'm like H it feels like a thing I always want to be true I like a thing I always want to be true I like a thing I always want to be true I kind of don't want models to be lying to kind of don't want models to be lying to kind of don't want models to be lying to people cuz if people are going to have people cuz if people are going to have people cuz if people are going to have like healthy relationships with anything like healthy relationships with anything like healthy relationships with anything it's kind of important yeah like I think it's kind of important yeah like I think it's kind of important yeah like I think that's easier if you always just like that's easier if you always just like that's easier if you always just like know exactly what the thing is that you know exactly what the thing is that you know exactly what the thing is that you relating to it doesn't solve everything relating to it doesn't solve everything relating to it doesn't solve everything but I think it helps quite anthropic may be the very company to anthropic may be the very company to develop a system that we definitively develop a system that we definitively develop a system that we definitively recognize as recognize as recognize as AGI and you very well might be the AGI and you very well might be the AGI and you very well might be the person that talks to it probably talks person that talks to it probably talks person that talks to it probably talks to it first well what would the

to it first well what would the to it first well what would the conversation contain like what would be conversation contain like what would be conversation contain like what would be your first question well it depends your first question well it depends your first question well it depends partly on like the kind of capability partly on like the kind of capability partly on like the kind of capability level of the model if you have something level of the model if you have something level of the model if you have something that is like capable in the same way that is like capable in the same way that is like capable in the same way that an extremely capable human is I that an extremely capable human is I that an extremely capable human is I imagine myself kind of interacting with imagine myself kind of interacting with imagine myself kind of interacting with it the same way that I do with an it the same way that I do with an it the same way that I do with an extremely capable human with the one extremely capable human with the one extremely capable human with the one difference that I'm probably going to be difference that I'm probably going to be difference that I'm probably going to be trying to like probe and understand its trying to like probe and understand its trying to like probe and understand its behaviors um but in many ways I'm like I behaviors um but in many ways I'm like I behaviors um but in many ways I'm like I can then just have like useful can then just have like useful can then just have like useful conversations with it you know so if I'm conversations with it you know so if I'm conversations with it you know so if I'm working on something as part of my working on something as part of my working on something as part of my research I can just be like oh like research I can just be like oh like research I can just be like oh like which I already find myself starting to which I already find myself starting to which I already find myself starting to do you know if I'm like oh I feel like do you know if I'm like oh I feel like do you know if I'm like oh I feel like there's this like thing in virtue ethics there's this like thing in virtue ethics there's this like thing in virtue ethics I can't quite remember the term like I can't quite remember the term like I can't quite remember the term like I'll use the model for things like that I'll use the model for things like that I'll use the model for things like that and so I could imagine that being more and so I could imagine that being more and so I could imagine that being more and more the case where you're just and more the case where you're just and more the case where you're just basically interacting with it much more basically interacting with it much more basically interacting with it much more like you would an incredibly smart colle like you would an incredibly smart colle like you would an incredibly smart colle colleague um and using it like for the colleague um and using it like for the colleague um and using it like for the kinds of work that you want to do as if kinds of work that you want to do as if kinds of work that you want to do as if you just had a collaborator who was like you just had a collaborator who was like you just had a collaborator who was like or you know the slightly horrifying or you know the slightly horrifying or you know the slightly horrifying thing about AI is like as soon as you thing about AI is like as soon as you thing about AI is like as soon as you have one collaborator you have a have one collaborator you have a have one collaborator you have a thousand collaborators if you can manage thousand collaborators if you can manage thousand collaborators if you can manage them enough but what if it's two times them enough but what if it's two times them enough but what if it's two times the smartest human on earth on that the smartest human on earth on that the smartest human on earth on that particular discipline yeah I guess particular discipline yeah I guess particular discipline yeah I guess you're really good at sort of probing you're really good at sort of probing you're really good at sort of probing claw um in a way that pushes its limits claw um in a way that pushes its limits claw um in a way that pushes its limits understanding where the limits are yep understanding where the limits are yep understanding where the limits are yep so I guess what would be a question you so I guess what would be a question you so I guess what would be a question you would ask to be like yeah this is Agi that's really hard because it feels Agi that's really hard because it feels like in order to it has to just be a like in order to it has to just be a like in order to it has to just be a series of questions like if there was series of questions like if there was series of questions like if there was just one question like you can train just one question like you can train just one question like you can train anything to answer one question

anything to answer one question anything to answer one question extremely well yeah um in fact you can extremely well yeah um in fact you can extremely well yeah um in fact you can probably train it to answer like you probably train it to answer like you probably train it to answer like you know 20 Questions extremely well like know 20 Questions extremely well like know 20 Questions extremely well like how long would you need to be locked in how long would you need to be locked in how long would you need to be locked in the room with an AGI to know this thing the room with an AGI to know this thing the room with an AGI to know this thing is Agi is Agi is Agi it's a hard question because part of me it's a hard question because part of me it's a hard question because part of me is like all of this just feels is like all of this just feels is like all of this just feels continuous like if you put me in a room continuous like if you put me in a room continuous like if you put me in a room for five minutes I'm like I just have for five minutes I'm like I just have for five minutes I'm like I just have high error bars you know I'm like and high error bars you know I'm like and high error bars you know I'm like and then it's just like maybe it's like both then it's just like maybe it's like both then it's just like maybe it's like both the the probability increases and the the the probability increases and the the the probability increases and the air bar decreases I think things that I air bar decreases I think things that I air bar decreases I think things that I can actually probe the edge of human can actually probe the edge of human can actually probe the edge of human knowledge of so I think this with knowledge of so I think this with knowledge of so I think this with philosophy a little bit sometimes when I philosophy a little bit sometimes when I philosophy a little bit sometimes when I ask the models philosophy questions I am ask the models philosophy questions I am ask the models philosophy questions I am like this is a question that I think no like this is a question that I think no like this is a question that I think no one has ever asked like it's maybe like one has ever asked like it's maybe like one has ever asked like it's maybe like right at the edge of like some right at the edge of like some right at the edge of like some literature that I know um and the models literature that I know um and the models literature that I know um and the models will just kind of like when they will just kind of like when they will just kind of like when they struggle with that when they struggle to struggle with that when they struggle to struggle with that when they struggle to come up with a kind of like novel like come up with a kind of like novel like come up with a kind of like novel like I'm like I know that there's like a I'm like I know that there's like a I'm like I know that there's like a novel argument here because I've just novel argument here because I've just novel argument here because I've just thought of it myself so maybe that's the thought of it myself so maybe that's the thought of it myself so maybe that's the thing where I'm like I've thought of a thing where I'm like I've thought of a thing where I'm like I've thought of a cool novel argument in this like Niche cool novel argument in this like Niche cool novel argument in this like Niche area and I'm going to just like probe area and I'm going to just like probe area and I'm going to just like probe you to see if you can come up with it you to see if you can come up with it you to see if you can come up with it and how much like prompting it takes to and how much like prompting it takes to and how much like prompting it takes to get you to come up with it and I think get you to come up with it and I think get you to come up with it and I think for some of these like really like uh for some of these like really like uh for some of these like really like uh right at the ede of human Knowledge right at the ede of human Knowledge right at the ede of human Knowledge Questions I'm like you could not in fact Questions I'm like you could not in fact Questions I'm like you could not in fact come up with the thing that I came up come up with the thing that I came up come up with the thing that I came up with I think if I just with I think if I just with I think if I just took something like that where I like I took something like that where I like I took something like that where I like I know a lot about an area and I came up know a lot about an area and I came up know a lot about an area and I came up with a novel issue or a novel like with a novel issue or a novel like with a novel issue or a novel like solution to a problem and I gave it to a solution to a problem and I gave it to a solution to a problem and I gave it to a model and it came up with that solution model and it came up with that solution model and it came up with that solution that would be a pretty moving moment for that would be a pretty moving moment for that would be a pretty moving moment for me because I would be like this is a me because I would be like this is a me because I would be like this is a case where no human has ever like it's

case where no human has ever like it's case where no human has ever like it's not and obviously we see these with this not and obviously we see these with this not and obviously we see these with this with like more kind of like you see with like more kind of like you see with like more kind of like you see novel Solutions all the time especially novel Solutions all the time especially novel Solutions all the time especially to like easier problems I think people to like easier problems I think people to like easier problems I think people overestimate you know novelty isn't like overestimate you know novelty isn't like overestimate you know novelty isn't like is completely different from anything is completely different from anything is completely different from anything ever happened it's just like this is it ever happened it's just like this is it ever happened it's just like this is it can be a variant of things that have can be a variant of things that have can be a variant of things that have happened um and still be novel but I happened um and still be novel but I happened um and still be novel but I think yeah if I saw like the the more I think yeah if I saw like the the more I think yeah if I saw like the the more I were to see like um completely like uh were to see like um completely like uh were to see like um completely like uh novel work from the models that that novel work from the models that that novel work from the models that that would be like and this is just going to would be like and this is just going to would be like and this is just going to feel iterative it's one of those things feel iterative it's one of those things feel iterative it's one of those things where it's there's never it's like you where it's there's never it's like you where it's there's never it's like you know people I think want there to be know people I think want there to be know people I think want there to be like a moment and I'm like I don't know like a moment and I'm like I don't know like a moment and I'm like I don't know like I think that there might just never like I think that there might just never like I think that there might just never be a moment it might just be that be a moment it might just be that be a moment it might just be that there's just like this continuous there's just like this continuous there's just like this continuous ramping up I I have a sense that there ramping up I I have a sense that there ramping up I I have a sense that there will be things that a model can say that will be things that a model can say that will be things that a model can say that convinces you this is very it's not like convinces you this is very it's not like convinces you this is very it's not like uh like I've talked to people who are uh like I've talked to people who are uh like I've talked to people who are like truly wise mhm like there you could like truly wise mhm like there you could like truly wise mhm like there you could just tell there's a lot of horsepower just tell there's a lot of horsepower just tell there's a lot of horsepower there yep and if you 10x that I don't there yep and if you 10x that I don't there yep and if you 10x that I don't know I just feel like there's words you know I just feel like there's words you know I just feel like there's words you could say maybe ask it to generate a could say maybe ask it to generate a could say maybe ask it to generate a poem mhm and poem mhm and poem mhm and the and the poemy generates you're like the and the poemy generates you're like the and the poemy generates you're like yeah okay yeah whatever you did there I yeah okay yeah whatever you did there I yeah okay yeah whatever you did there I don't think a human can do that I think don't think a human can do that I think don't think a human can do that I think it has to be something that I can verify it has to be something that I can verify it has to be something that I can verify is like actually really good though is like actually really good though is like actually really good though that's why I think these questions that that's why I think these questions that that's why I think these questions that are like where I'm like oh this is like are like where I'm like oh this is like are like where I'm like oh this is like you know like you know sometimes it's

you know like you know sometimes it's you know like you know sometimes it's just like I'll come up with say a just like I'll come up with say a just like I'll come up with say a concrete counter example to like an concrete counter example to like an concrete counter example to like an argument or something like that I'm sure argument or something like that I'm sure argument or something like that I'm sure like with like it it would be like if like with like it it would be like if like with like it it would be like if you're a mathematician you had a novel you're a mathematician you had a novel you're a mathematician you had a novel proof I think and you just gave it the proof I think and you just gave it the proof I think and you just gave it the problem and you saw it and you're this problem and you saw it and you're this problem and you saw it and you're this proof is genuinely novel like there's no proof is genuinely novel like there's no proof is genuinely novel like there's no one has ever done you actually have to one has ever done you actually have to one has ever done you actually have to do a lot of things to like come up with do a lot of things to like come up with do a lot of things to like come up with this um you know I had to sit and think this um you know I had to sit and think this um you know I had to sit and think about it for months or something and about it for months or something and about it for months or something and then if you saw the model successfully then if you saw the model successfully then if you saw the model successfully do that I think you would just be like I do that I think you would just be like I do that I think you would just be like I can verify that this is correct it is can verify that this is correct it is can verify that this is correct it is like it is a sign that you have like it is a sign that you have like it is a sign that you have generalized from your training like you generalized from your training like you generalized from your training like you didn't just see this somewhere because I didn't just see this somewhere because I didn't just see this somewhere because I just came up with it myself and you were just came up with it myself and you were just came up with it myself and you were able to like replicate that um that's able to like replicate that um that's able to like replicate that um that's the kind of thing where I'm like for the kind of thing where I'm like for the kind of thing where I'm like for me the closer the more that models like me the closer the more that models like me the closer the more that models like can do things like that the more I would can do things like that the more I would can do things like that the more I would be like oh this is like uh very real cuz be like oh this is like uh very real cuz be like oh this is like uh very real cuz then I can I don't know I can like then I can I don't know I can like then I can I don't know I can like verify that that's like extremely verify that that's like extremely verify that that's like extremely extremely capable you've interacted with extremely capable you've interacted with extremely capable you've interacted with AI a lot what do you think makes humans AI a lot what do you think makes humans AI a lot what do you think makes humans special oh good question maybe in a way that the question maybe in a way that the universe is much better off that we're universe is much better off that we're universe is much better off that we're in it and that we should definitely in it and that we should definitely in it and that we should definitely survive and spread throughout the survive and spread throughout the survive and spread throughout the Universe yeah it's interesting because I Universe yeah it's interesting because I Universe yeah it's interesting because I think like people focus so much on think like people focus so much on think like people focus so much on intelligence especially with models look intelligence especially with models look intelligence especially with models look intelligence is important because of intelligence is important because of intelligence is important because of what it does like it's very useful it what it does like it's very useful it what it does like it's very useful it does a lot of things in the world and does a lot of things in the world and does a lot of things in the world and I'm like you know you can imagine a I'm like you know you can imagine a I'm like you know you can imagine a world where like height or strength world where like height or strength world where like height or strength would have played this role and I'm like

would have played this role and I'm like would have played this role and I'm like it's just a trait like that I'm like it's just a trait like that I'm like it's just a trait like that I'm like it's not intrinsically valuable it's it's not intrinsically valuable it's it's not intrinsically valuable it's it's valuable because of what it does I it's valuable because of what it does I it's valuable because of what it does I think for the most part um the things think for the most part um the things think for the most part um the things that feel you know I'm like that feel you know I'm like that feel you know I'm like I mean personally I'm just like I think I mean personally I'm just like I think I mean personally I'm just like I think humans and like life in general is humans and like life in general is humans and like life in general is extremely magical um we almost like to extremely magical um we almost like to extremely magical um we almost like to the degree that I you know I don't know the degree that I you know I don't know the degree that I you know I don't know like not everyone agrees with this I'm like not everyone agrees with this I'm like not everyone agrees with this I'm flagging but um you know we have this flagging but um you know we have this flagging but um you know we have this like whole universe and there's like all like whole universe and there's like all like whole universe and there's like all of these objects you know there's like of these objects you know there's like of these objects you know there's like beautiful stars and there's like beautiful stars and there's like beautiful stars and there's like galaxies and then I don't know I'm just galaxies and then I don't know I'm just galaxies and then I don't know I'm just like on this planet there are these like on this planet there are these like on this planet there are these creatures that have this like ability to creatures that have this like ability to creatures that have this like ability to observe that like uh and they are like observe that like uh and they are like observe that like uh and they are like seeing it they are experiencing it and seeing it they are experiencing it and seeing it they are experiencing it and I'm just like that if you try to explain I'm just like that if you try to explain I'm just like that if you try to explain like I'm I imagine trying to explain to like I'm I imagine trying to explain to like I'm I imagine trying to explain to like I don't know someone for some like I don't know someone for some like I don't know someone for some reason they they've never encountered reason they they've never encountered reason they they've never encountered the world or our science or anything and the world or our science or anything and the world or our science or anything and I think that nothing is that like I think that nothing is that like I think that nothing is that like everything you know like all of our everything you know like all of our everything you know like all of our physics and everything in the world it's physics and everything in the world it's physics and everything in the world it's all extremely exciting but then you say all extremely exciting but then you say all extremely exciting but then you say oh and plus there's this thing that it oh and plus there's this thing that it oh and plus there's this thing that it is to be a thing and observe in the is to be a thing and observe in the is to be a thing and observe in the world and and you see this like inner world and and you see this like inner world and and you see this like inner Cinema and I think they would be like Cinema and I think they would be like Cinema and I think they would be like hang on wait pause you just said hang on wait pause you just said hang on wait pause you just said something that like is kind of wild something that like is kind of wild something that like is kind of wild sounding sounding sounding um and so I'm like we have this like um and so I'm like we have this like um and so I'm like we have this like ability to like experience the world um ability to like experience the world um ability to like experience the world um we feel pleasure we feel suffering we we feel pleasure we feel suffering we we feel pleasure we feel suffering we feel like a lot of like complex things feel like a lot of like complex things feel like a lot of like complex things and so yeah and maybe this is also why I and so yeah and maybe this is also why I and so yeah and maybe this is also why I think you know I also like hear a lot think you know I also like hear a lot think you know I also like hear a lot about animals for example because I about animals for example because I about animals for example because I think they probably share this with us think they probably share this with us think they probably share this with us um so I think that like the things that

um so I think that like the things that um so I think that like the things that make humans special in so far as like I make humans special in so far as like I make humans special in so far as like I care about humans is probably more like care about humans is probably more like care about humans is probably more like their ability to to feel and experience their ability to to feel and experience their ability to to feel and experience than it is like them having these like than it is like them having these like than it is like them having these like functional useful traits yeah to to feel functional useful traits yeah to to feel functional useful traits yeah to to feel and experience the beauty in the world and experience the beauty in the world and experience the beauty in the world yeah to look at the yeah to look at the yeah to look at the stars I hope there's other civiliz alien stars I hope there's other civiliz alien stars I hope there's other civiliz alien civilizations out there but if we're it civilizations out there but if we're it civilizations out there but if we're it it's a pretty good uh it's a pretty good it's a pretty good uh it's a pretty good it's a pretty good uh it's a pretty good thing and that they're having a good thing and that they're having a good thing and that they're having a good time they're having a good time watching time they're having a good time watching time they're having a good time watching us yeah well um thank you for this good us yeah well um thank you for this good us yeah well um thank you for this good time of a conversation and for the work time of a conversation and for the work time of a conversation and for the work you're doing and for helping make uh you're doing and for helping make uh you're doing and for helping make uh Claude a great conversational partner Claude a great conversational partner Claude a great conversational partner and thank you for talking today yeah and thank you for talking today yeah and thank you for talking today yeah thanks for talking thanks for listening thanks for talking thanks for listening thanks for talking thanks for listening to this conversation with Amanda ascal to this conversation with Amanda ascal to this conversation with Amanda ascal and now dear friends here's Chris and now dear friends here's Chris and now dear friends here's Chris Ola can you Ola can you Ola can you describe this fascinating field of describe this fascinating field of describe this fascinating field of mechanistic interpretability AKA Mech mechanistic interpretability AKA Mech mechanistic interpretability AKA Mech interp the history of the field and interp the history of the field and interp the history of the field and where is the today I think one useful where is the today I think one useful where is the today I think one useful way to think about neural networks is way to think about neural networks is way to think about neural networks is that we don't we don't program we don't that we don't we don't program we don't that we don't we don't program we don't make them we we kind of we grow them you make them we we kind of we grow them you make them we we kind of we grow them you know we have these neural network know we have these neural network know we have these neural network architectures that we design and we have architectures that we design and we have architectures that we design and we have these loss objectives that we that we we these loss objectives that we that we we these loss objectives that we that we we create and the neural network create and the neural network create and the neural network architecture it's kind of like a architecture it's kind of like a architecture it's kind of like a scaffold that the circuits grow on um scaffold that the circuits grow on um scaffold that the circuits grow on um and they sort of you know it starts off and they sort of you know it starts off and they sort of you know it starts off with some kind of random you know random with some kind of random you know random with some kind of random you know random things and it grows and it's almost like things and it grows and it's almost like things and it grows and it's almost like the the objective that we train for is the the objective that we train for is the the objective that we train for is this light um and so we create the this light um and so we create the this light um and so we create the scaffold that it grows on and we create

scaffold that it grows on and we create scaffold that it grows on and we create the you know the light that it grows the you know the light that it grows the you know the light that it grows towards but the thing that we actually towards but the thing that we actually towards but the thing that we actually create it's it's it's this almost create it's it's it's this almost create it's it's it's this almost biological biological biological you know entity or organism that we're you know entity or organism that we're you know entity or organism that we're that we're studying um and so it's very that we're studying um and so it's very that we're studying um and so it's very very different from any kind of regular very different from any kind of regular very different from any kind of regular software engineering um because at the software engineering um because at the software engineering um because at the end of the day we end up with this end of the day we end up with this end of the day we end up with this artifact that can do all these amazing artifact that can do all these amazing artifact that can do all these amazing things it can you know write essays and things it can you know write essays and things it can you know write essays and translate and you know understand images translate and you know understand images translate and you know understand images it can do all these things that we have it can do all these things that we have it can do all these things that we have no idea how to directly create a no idea how to directly create a no idea how to directly create a computer program to do and it can do computer program to do and it can do computer program to do and it can do that because we we grew it we didn't we that because we we grew it we didn't we that because we we grew it we didn't we didn't write it we didn't create it and didn't write it we didn't create it and didn't write it we didn't create it and so then that leaves open this question so then that leaves open this question so then that leaves open this question at the end which is what the hell is at the end which is what the hell is at the end which is what the hell is going on inside these systems um and going on inside these systems um and going on inside these systems um and that you know is uh you know to me um a that you know is uh you know to me um a that you know is uh you know to me um a really deep and exciting question it's really deep and exciting question it's really deep and exciting question it's you know a a really exciting scientific you know a a really exciting scientific you know a a really exciting scientific question to me it's it's it's sort of is question to me it's it's it's sort of is question to me it's it's it's sort of is like the question that is is just like the question that is is just like the question that is is just screaming out it's calling out for us to screaming out it's calling out for us to screaming out it's calling out for us to go and answer it when we talk about Nal go and answer it when we talk about Nal go and answer it when we talk about Nal networks and I think it's also a very networks and I think it's also a very networks and I think it's also a very deep question for safety reasons so and deep question for safety reasons so and deep question for safety reasons so and mechanistic interpretability I guess is mechanistic interpretability I guess is mechanistic interpretability I guess is closer to maybe neurobiology yeah yeah I closer to maybe neurobiology yeah yeah I closer to maybe neurobiology yeah yeah I think that's right so maybe to give an think that's right so maybe to give an think that's right so maybe to give an example of the kind of thing that has example of the kind of thing that has example of the kind of thing that has been done that I wouldn't consider to be been done that I wouldn't consider to be been done that I wouldn't consider to be mechanistic inability there was um for a mechanistic inability there was um for a mechanistic inability there was um for a long time a lot of work on saliency maps long time a lot of work on saliency maps long time a lot of work on saliency maps where you would take an image and you where you would take an image and you where you would take an image and you try to say you know the model thinks try to say you know the model thinks try to say you know the model thinks this image is a dog what part of the this image is a dog what part of the this image is a dog what part of the image made it think that it's a dog um image made it think that it's a dog um image made it think that it's a dog um and you know that tells you maybe and you know that tells you maybe and you know that tells you maybe something about the model if you can something about the model if you can something about the model if you can come up with a principled version of come up with a principled version of come up with a principled version of that um but it doesn't really tell you

that um but it doesn't really tell you that um but it doesn't really tell you like what algorithms are running in the like what algorithms are running in the like what algorithms are running in the model how was the model actually making model how was the model actually making model how was the model actually making that decision maybe it's telling you that decision maybe it's telling you that decision maybe it's telling you something about what was important to it something about what was important to it something about what was important to it if you if you can make that meth work if you if you can make that meth work if you if you can make that meth work but it it isn't telling you you know but it it isn't telling you you know but it it isn't telling you you know what are what are the algorithms that what are what are the algorithms that what are what are the algorithms that are running how is it that this the are running how is it that this the are running how is it that this the system is able to do this thing that we system is able to do this thing that we system is able to do this thing that we no one knew how to do and so I guess we no one knew how to do and so I guess we no one knew how to do and so I guess we started using the term mechanistic started using the term mechanistic started using the term mechanistic inability to try to sort of draw that inability to try to sort of draw that inability to try to sort of draw that that divide or to distinguish ourselves that divide or to distinguish ourselves that divide or to distinguish ourselves and the work that we were doing in some and the work that we were doing in some and the work that we were doing in some ways from from some of these other ways from from some of these other ways from from some of these other things and I think since then it's things and I think since then it's things and I think since then it's become this sort of umbrella term for um become this sort of umbrella term for um become this sort of umbrella term for um you know pretty wide variety of work but you know pretty wide variety of work but you know pretty wide variety of work but I'd say that the things that that are I'd say that the things that that are I'd say that the things that that are kind of distinctive are I think a this kind of distinctive are I think a this kind of distinctive are I think a this this focus on we really want to get at this focus on we really want to get at this focus on we really want to get at you know the mechanisms we want to get you know the mechanisms we want to get you know the mechanisms we want to get at the algorithms um you know if you at the algorithms um you know if you at the algorithms um you know if you think of if you think of neural networks think of if you think of neural networks think of if you think of neural networks as being like a computer program um then as being like a computer program um then as being like a computer program um then the weights are kind of like a binary the weights are kind of like a binary the weights are kind of like a binary computer program and we'd like to computer program and we'd like to computer program and we'd like to reverse engineer those weights and reverse engineer those weights and reverse engineer those weights and figure out what algorithms are running figure out what algorithms are running figure out what algorithms are running so okay I think one way you might think so okay I think one way you might think so okay I think one way you might think of trying to understand a neural network of trying to understand a neural network of trying to understand a neural network is that it's it's kind of like a we have is that it's it's kind of like a we have is that it's it's kind of like a we have this compiled computer program and the this compiled computer program and the this compiled computer program and the weights of the neural network are are weights of the neural network are are weights of the neural network are are the binary um and when the neural the binary um and when the neural the binary um and when the neural network runs that's that's the network runs that's that's the network runs that's that's the activations um and our our goal is activations um and our our goal is activations um and our our goal is ultimately to go and understand and ultimately to go and understand and ultimately to go and understand and understand these weights and so you know understand these weights and so you know understand these weights and so you know the project mechanistic inability is to the project mechanistic inability is to the project mechanistic inability is to somehow figure out how do these weights somehow figure out how do these weights somehow figure out how do these weights correspond to correspond to correspond to algorithms um and in order to do that algorithms um and in order to do that algorithms um and in order to do that you also have to understand the you also have to understand the you also have to understand the activations because it's sort of the activations because it's sort of the activations because it's sort of the activations are like the memory and if activations are like the memory and if activations are like the memory and if you if you imagine reverse engineering a you if you imagine reverse engineering a you if you imagine reverse engineering a computer program um and you have the computer program um and you have the computer program um and you have the binary instructions you know in order to binary instructions you know in order to binary instructions you know in order to understand what what a particular understand what what a particular understand what what a particular instruction means you need to know what instruction means you need to know what instruction means you need to know what me what what is stored in the memory

me what what is stored in the memory me what what is stored in the memory that it's operating on and so those two that it's operating on and so those two that it's operating on and so those two things are very intertwined so things are very intertwined so things are very intertwined so mechanistic interpret tends to be mechanistic interpret tends to be mechanistic interpret tends to be interested in both of those things now interested in both of those things now interested in both of those things now you there's a lot of work that's that's you there's a lot of work that's that's you there's a lot of work that's that's interested in in in those things um interested in in in those things um interested in in in those things um especially the you know there's all this especially the you know there's all this especially the you know there's all this work on probing which you might see as work on probing which you might see as work on probing which you might see as part of being mechanistic interality part of being mechanistic interality part of being mechanistic interality although it's you know again it's just a although it's you know again it's just a although it's you know again it's just a broad term and and not everyone who does broad term and and not everyone who does broad term and and not everyone who does that work would identify as doing Mech I that work would identify as doing Mech I that work would identify as doing Mech I think the thing that is maybe a little think the thing that is maybe a little think the thing that is maybe a little bit distinctive to the the vibe of bit distinctive to the the vibe of bit distinctive to the the vibe of mechant turp is I think people tend mechant turp is I think people tend mechant turp is I think people tend working in the space tend to think of working in the space tend to think of working in the space tend to think of neural networks as well maybe one way to neural networks as well maybe one way to neural networks as well maybe one way to said is that greent descent is smarter said is that greent descent is smarter said is that greent descent is smarter than you that you know uh and gradient than you that you know uh and gradient than you that you know uh and gradient descent is is actually really great the descent is is actually really great the descent is is actually really great the whole reason that we're understanding whole reason that we're understanding whole reason that we're understanding these models is because we didn't know these models is because we didn't know these models is because we didn't know how to write them in the first place the how to write them in the first place the how to write them in the first place the gradient descent comes up with better gradient descent comes up with better gradient descent comes up with better Solutions than us and so um I think that Solutions than us and so um I think that Solutions than us and so um I think that maybe another thing about mechant turp maybe another thing about mechant turp maybe another thing about mechant turp is sort of having almost a kind of is sort of having almost a kind of is sort of having almost a kind of humility that we won't guess at prior humility that we won't guess at prior humility that we won't guess at prior what's going on inside the model and we what's going on inside the model and we what's going on inside the model and we have to have the sort of bottom up have to have the sort of bottom up have to have the sort of bottom up approach where we don't really assume approach where we don't really assume approach where we don't really assume you know we don't assume that we should you know we don't assume that we should you know we don't assume that we should look for a particular thing and that look for a particular thing and that look for a particular thing and that will be there and that's how it works will be there and that's how it works will be there and that's how it works but instead we look from the bottom up but instead we look from the bottom up but instead we look from the bottom up and discover what happens to exist in and discover what happens to exist in and discover what happens to exist in these models and study them that way but these models and study them that way but these models and study them that way but you know the very fact that it's you know the very fact that it's you know the very fact that it's possible to do and as you and others possible to do and as you and others possible to do and as you and others have shown over time you know things have shown over time you know things have shown over time you know things like like like universality universality universality that the wisdom of The gradian Descent that the wisdom of The gradian Descent that the wisdom of The gradian Descent creates features and circus creates creates features and circus creates creates features and circus creates things universally across different things universally across different things universally across different kinds of networks that are useful and kinds of networks that are useful and kinds of networks that are useful and that makes the whole field possible yeah that makes the whole field possible yeah that makes the whole field possible yeah so this is actually is indeed a a really so this is actually is indeed a a really so this is actually is indeed a a really remarkable and exciting thing where it

remarkable and exciting thing where it remarkable and exciting thing where it does seem like at least to some extent does seem like at least to some extent does seem like at least to some extent you know the same the same elements the you know the same the same elements the you know the same the same elements the same the same features and circuits form same the same features and circuits form same the same features and circuits form again and again um you know you can look again and again um you know you can look again and again um you know you can look at every Vision model and you'll find at every Vision model and you'll find at every Vision model and you'll find curve detectors and you'll find high low curve detectors and you'll find high low curve detectors and you'll find high low frequency detectors um and in fact frequency detectors um and in fact frequency detectors um and in fact there's some some reason to think that there's some some reason to think that there's some some reason to think that the same things form across you know the same things form across you know the same things form across you know biological neural networks and biological neural networks and biological neural networks and artificial neural networks so a famous artificial neural networks so a famous artificial neural networks so a famous example is Vision Vision models in in example is Vision Vision models in in example is Vision Vision models in in the early layers they have Gabor filters the early layers they have Gabor filters the early layers they have Gabor filters and there's you know Gabor filters are and there's you know Gabor filters are and there's you know Gabor filters are something that neuroscientists are something that neuroscientists are something that neuroscientists are interested and have thought a lot about interested and have thought a lot about interested and have thought a lot about we find curved detectors in these models we find curved detectors in these models we find curved detectors in these models curve detectors are also found in curve detectors are also found in curve detectors are also found in monkeys we discover these high low monkeys we discover these high low monkeys we discover these high low frequency detectors and then um some frequency detectors and then um some frequency detectors and then um some followup work went and discovered them followup work went and discovered them followup work went and discovered them um in rats um or mice um so they were um in rats um or mice um so they were um in rats um or mice um so they were found first in artificial neural found first in artificial neural found first in artificial neural networks and then found in biological networks and then found in biological networks and then found in biological neural networks um you know this really neural networks um you know this really neural networks um you know this really famous result on like grandmother famous result on like grandmother famous result on like grandmother neurons or the um the Haley Berry neuron neurons or the um the Haley Berry neuron neurons or the um the Haley Berry neuron from quiroa at all and we found very from quiroa at all and we found very from quiroa at all and we found very similar things in in Vision models where similar things in in Vision models where similar things in in Vision models where this is while I was still at open Ai and this is while I was still at open Ai and this is while I was still at open Ai and I I was looking at their clip model um I I was looking at their clip model um I I was looking at their clip model um and you find um these neurons that and you find um these neurons that and you find um these neurons that respond to the same entities in images respond to the same entities in images respond to the same entities in images and also to give a concrete example and also to give a concrete example and also to give a concrete example there we found that there was a Donald there we found that there was a Donald there we found that there was a Donald Trump n for some reason I guess Everyone Trump n for some reason I guess Everyone Trump n for some reason I guess Everyone likes to talk about Donald Trump and and likes to talk about Donald Trump and and likes to talk about Donald Trump and and Donald Trump was very prominent was was Donald Trump was very prominent was was Donald Trump was very prominent was was very a very Hot Topic at that time so very a very Hot Topic at that time so very a very Hot Topic at that time so every every neural network that we every every neural network that we every every neural network that we looked at we would find a dedicated looked at we would find a dedicated looked at we would find a dedicated neuron for Donald Trump um that was the neuron for Donald Trump um that was the neuron for Donald Trump um that was the only person who had always had a only person who had always had a only person who had always had a dedicated nuron um you know sometimes dedicated nuron um you know sometimes dedicated nuron um you know sometimes you'd have an Obama nuran sometimes you'd have an Obama nuran sometimes you'd have an Obama nuran sometimes you'd have a Clinton Nan but uh Trump you'd have a Clinton Nan but uh Trump you'd have a Clinton Nan but uh Trump always had a dedicate so it responds to always had a dedicate so it responds to always had a dedicate so it responds to you know pictures of his face and the you know pictures of his face and the you know pictures of his face and the ward Trump like all these things right

ward Trump like all these things right ward Trump like all these things right um and so it's it's not responding to a um and so it's it's not responding to a um and so it's it's not responding to a particular example or like it's not just particular example or like it's not just particular example or like it's not just responding to his face it's it's responding to his face it's it's responding to his face it's it's abstracting over this General concept abstracting over this General concept abstracting over this General concept right so in any case that's very similar right so in any case that's very similar right so in any case that's very similar to these qu results so there this to these qu results so there this to these qu results so there this evidence that these that this fomen of evidence that these that this fomen of evidence that these that this fomen of universality the same things form across universality the same things form across universality the same things form across both artificial and and natural neural both artificial and and natural neural both artificial and and natural neural networks that's that's a pretty amazing networks that's that's a pretty amazing networks that's that's a pretty amazing thing if that's true um you know it thing if that's true um you know it thing if that's true um you know it suggests that um well I think the thing suggests that um well I think the thing suggests that um well I think the thing that it suggests is the gradi scent is that it suggests is the gradi scent is that it suggests is the gradi scent is sort of finding you know the right ways sort of finding you know the right ways sort of finding you know the right ways to cut things apart in some sense that to cut things apart in some sense that to cut things apart in some sense that many systems converge on and and many many systems converge on and and many many systems converge on and and many different neural networks architectures different neural networks architectures different neural networks architectures converge on that there's there's some converge on that there's there's some converge on that there's there's some natural set of you know there's some set natural set of you know there's some set natural set of you know there's some set of abstractions that are a very natural of abstractions that are a very natural of abstractions that are a very natural way to cut apart the problem and that a way to cut apart the problem and that a way to cut apart the problem and that a lot of systems are going to converge on lot of systems are going to converge on lot of systems are going to converge on um that would be my my kind of uh you um that would be my my kind of uh you um that would be my my kind of uh you know I don't know anything about know I don't know anything about know I don't know anything about Neuroscience this is this is just my my Neuroscience this is this is just my my Neuroscience this is this is just my my kind of wild speculation from what we've kind of wild speculation from what we've kind of wild speculation from what we've seen yeah that would be beautiful if seen yeah that would be beautiful if seen yeah that would be beautiful if it's sort of agnostic to the it's sort of agnostic to the it's sort of agnostic to the medium of uh of the model that's used to medium of uh of the model that's used to medium of uh of the model that's used to form the representation yeah yeah and form the representation yeah yeah and form the representation yeah yeah and it's you know it's um a a kind of a wild it's you know it's um a a kind of a wild it's you know it's um a a kind of a wild speculation based you know we only have speculation based you know we only have speculation based you know we only have some a few data points justest this but some a few data points justest this but some a few data points justest this but you know it it does seem like there's um you know it it does seem like there's um you know it it does seem like there's um there's some sense in which the same there's some sense in which the same there's some sense in which the same things form again again and again and things form again again and again and things form again again and again and again both in certainly in natural again both in certainly in natural again both in certainly in natural neural networks and and also neural networks and and also neural networks and and also artificially or in biologically and the artificially or in biologically and the artificially or in biologically and the intuition behind that would be that you intuition behind that would be that you intuition behind that would be that you know where in order to be useful in know where in order to be useful in know where in order to be useful in understanding the real world you need understanding the real world you need understanding the real world you need all the same kind of stuff yeah well if all the same kind of stuff yeah well if all the same kind of stuff yeah well if we pick I don't know like the idea of a we pick I don't know like the idea of a we pick I don't know like the idea of a dog right like you know there's some dog right like you know there's some dog right like you know there's some sense in which the idea of a dog is like

sense in which the idea of a dog is like sense in which the idea of a dog is like an a a natural category in the universe an a a natural category in the universe an a a natural category in the universe or something like this right like you or something like this right like you or something like this right like you know know know uh uh there's there's some reason it's uh uh there's there's some reason it's uh uh there's there's some reason it's it's not just like a weird Quirk of like it's not just like a weird Quirk of like it's not just like a weird Quirk of like how humans Factor you know think about how humans Factor you know think about how humans Factor you know think about the world that we have this concept of a the world that we have this concept of a the world that we have this concept of a dog it's it's in some sense or or like dog it's it's in some sense or or like dog it's it's in some sense or or like if you have the idea of a line like if you have the idea of a line like if you have the idea of a line like there's you know like look around us you there's you know like look around us you there's you know like look around us you know the you know there are lines you know the you know there are lines you know the you know there are lines you know it's sort of the simplest way to know it's sort of the simplest way to know it's sort of the simplest way to understand this room in some sense is to understand this room in some sense is to understand this room in some sense is to have the idea of a line and so um I have the idea of a line and so um I have the idea of a line and so um I think that that would be my instinct for think that that would be my instinct for think that that would be my instinct for why this happens yeah you need a curved why this happens yeah you need a curved why this happens yeah you need a curved line you know to understand a circle and line you know to understand a circle and line you know to understand a circle and you need all those shapes to understand you need all those shapes to understand you need all those shapes to understand bigger things and yeah it's a hierarchy bigger things and yeah it's a hierarchy bigger things and yeah it's a hierarchy of Concepts that are formed yeah and of Concepts that are formed yeah and of Concepts that are formed yeah and like maybe there are ways to go and like maybe there are ways to go and like maybe there are ways to go and describe you know images without describe you know images without describe you know images without reference to those things right but reference to those things right but reference to those things right but they're not the simplest way or the most they're not the simplest way or the most they're not the simplest way or the most economical way or something like this economical way or something like this economical way or something like this and so systems converge to these um and so systems converge to these um and so systems converge to these um these these strategies would would be my these these strategies would would be my these these strategies would would be my my wild wild hypothesis can you talk my wild wild hypothesis can you talk my wild wild hypothesis can you talk through some of the building blocks that through some of the building blocks that through some of the building blocks that we've been referencing of features and we've been referencing of features and we've been referencing of features and circuits so I think you first described circuits so I think you first described circuits so I think you first described them in uh 2020 paper zoom in and them in uh 2020 paper zoom in and them in uh 2020 paper zoom in and introduction to circuits absolutely so introduction to circuits absolutely so introduction to circuits absolutely so um maybe I'll start by just describing um maybe I'll start by just describing um maybe I'll start by just describing some phenomena and then we can sort of some phenomena and then we can sort of some phenomena and then we can sort of build to the idea of features and build to the idea of features and build to the idea of features and circuits so um if you spent like quite a circuits so um if you spent like quite a circuits so um if you spent like quite a few years maybe maybe like five years to few years maybe maybe like five years to few years maybe maybe like five years to some extent um with other things some extent um with other things some extent um with other things studying this one particular model studying this one particular model studying this one particular model Inception V1 um which is this one Vision Inception V1 um which is this one Vision Inception V1 um which is this one Vision model it was um state-ofthe-art in 2015 model it was um state-ofthe-art in 2015 model it was um state-ofthe-art in 2015 um and uh uh you know very much not um and uh uh you know very much not um and uh uh you know very much not state-ofthe-art anymore um and it has state-ofthe-art anymore um and it has state-ofthe-art anymore um and it has you know maybe about 10,000 neurons and

you know maybe about 10,000 neurons and you know maybe about 10,000 neurons and and I spent a lot of time looking at the and I spent a lot of time looking at the and I spent a lot of time looking at the 10,000 neurons 10,000 neurons 10,000 neurons odd neurons of of inception V1 odd neurons of of inception V1 odd neurons of of inception V1 um and one of the interesting things is um and one of the interesting things is um and one of the interesting things is you know there are lots of neurons that you know there are lots of neurons that you know there are lots of neurons that don't have some obvious intal meaning don't have some obvious intal meaning don't have some obvious intal meaning but there's a lot of neurons on but there's a lot of neurons on but there's a lot of neurons on Inception V1 that do have really clean Inception V1 that do have really clean Inception V1 that do have really clean intal meanings um so you find neurons intal meanings um so you find neurons intal meanings um so you find neurons that just really do seem to detect that just really do seem to detect that just really do seem to detect curves and you find neurons that really curves and you find neurons that really curves and you find neurons that really do seem to detect cars and um car wheels do seem to detect cars and um car wheels do seem to detect cars and um car wheels and car windows and you know floppy ears and car windows and you know floppy ears and car windows and you know floppy ears of dogs and dogs with long snouts facing of dogs and dogs with long snouts facing of dogs and dogs with long snouts facing to the right and dogs with Longs Nots to the right and dogs with Longs Nots to the right and dogs with Longs Nots facing to the left and you know facing to the left and you know facing to the left and you know different kinds of far and there's different kinds of far and there's different kinds of far and there's there's sort of this whole beautiful there's sort of this whole beautiful there's sort of this whole beautiful Edge detectors line detectors color Edge detectors line detectors color Edge detectors line detectors color contrast detectors um these beautiful contrast detectors um these beautiful contrast detectors um these beautiful things we call high low frequency things we call high low frequency things we call high low frequency detectors you know I think looking at I detectors you know I think looking at I detectors you know I think looking at I sort of felt like a biologist you know sort of felt like a biologist you know sort of felt like a biologist you know you just you're looking at at this sort you just you're looking at at this sort you just you're looking at at this sort of new world of proteins and you're of new world of proteins and you're of new world of proteins and you're discovering all these these different discovering all these these different discovering all these these different proteins that proteins that proteins that interact um so one way you could try to interact um so one way you could try to interact um so one way you could try to understand these models is in terms of understand these models is in terms of understand these models is in terms of neurons you could try to be like oh you neurons you could try to be like oh you neurons you could try to be like oh you know there's a dog detecting neuron and know there's a dog detecting neuron and know there's a dog detecting neuron and um here's a car detecting neuron and it um here's a car detecting neuron and it um here's a car detecting neuron and it turns out you can actually ask how those turns out you can actually ask how those turns out you can actually ask how those connect together so you can go and say connect together so you can go and say connect together so you can go and say oh you know I have this car detecting on oh you know I have this car detecting on oh you know I have this car detecting on how was it built and it turns out in the how was it built and it turns out in the how was it built and it turns out in the previous layer it's connected really previous layer it's connected really previous layer it's connected really strongly to a window detector and a strongly to a window detector and a strongly to a window detector and a wheel detector and a sort of car body wheel detector and a sort of car body wheel detector and a sort of car body detector and it looks for the window detector and it looks for the window detector and it looks for the window above the car and the wheels below and above the car and the wheels below and above the car and the wheels below and the car chrome sort of in the middle the car chrome sort of in the middle the car chrome sort of in the middle sort of everywhere but especially on the sort of everywhere but especially on the sort of everywhere but especially on the lower part um and that's sort of a lower part um and that's sort of a lower part um and that's sort of a recipe for a car right like that is you recipe for a car right like that is you recipe for a car right like that is you know earlier we said the thing we wanted know earlier we said the thing we wanted know earlier we said the thing we wanted from mechor was to get algorithms to go from mechor was to get algorithms to go from mechor was to get algorithms to go and get you know ask what is the the and get you know ask what is the the and get you know ask what is the the algorithm that runs well here we're just

algorithm that runs well here we're just algorithm that runs well here we're just looking at the weights of the N Network looking at the weights of the N Network looking at the weights of the N Network reading off this kind of recipe for reading off this kind of recipe for reading off this kind of recipe for detecting cars it's a very simple crude detecting cars it's a very simple crude detecting cars it's a very simple crude recipe but it's it's there and so we recipe but it's it's there and so we recipe but it's it's there and so we call that a circuit this this connection call that a circuit this this connection call that a circuit this this connection well okay so the the problem is that not well okay so the the problem is that not well okay so the the problem is that not all of the neurons um are interpal and all of the neurons um are interpal and all of the neurons um are interpal and there there's reason to think um we can there there's reason to think um we can there there's reason to think um we can get into this more later that there's get into this more later that there's get into this more later that there's this this superos hypothesis there this this superos hypothesis there this this superos hypothesis there reason to think that sometimes the right reason to think that sometimes the right reason to think that sometimes the right unit to analyze things in terms of um is unit to analyze things in terms of um is unit to analyze things in terms of um is combinations of neurons so sometimes combinations of neurons so sometimes combinations of neurons so sometimes it's not that there's a single neuron it's not that there's a single neuron it's not that there's a single neuron that represents say a car um but it that represents say a car um but it that represents say a car um but it actually turns that after you detect the actually turns that after you detect the actually turns that after you detect the car the model sort of hides a little bit car the model sort of hides a little bit car the model sort of hides a little bit of the car in the following layer and a of the car in the following layer and a of the car in the following layer and a bunch of a bunch of dog detectors why is bunch of a bunch of dog detectors why is bunch of a bunch of dog detectors why is it doing that well you know maybe it it doing that well you know maybe it it doing that well you know maybe it just doesn't want to do that much work just doesn't want to do that much work just doesn't want to do that much work on on on on cars at that point and you on on on on cars at that point and you on on on on cars at that point and you know it's sort of storing it away to go know it's sort of storing it away to go know it's sort of storing it away to go and um uh so it turns out then that the and um uh so it turns out then that the and um uh so it turns out then that the sort of subtle pattern of you know sort of subtle pattern of you know sort of subtle pattern of you know there's all these neurons that you think there's all these neurons that you think there's all these neurons that you think are dog detectors and maybe they're are dog detectors and maybe they're are dog detectors and maybe they're primarily that but they all a little bit primarily that but they all a little bit primarily that but they all a little bit contribute to representing a car um in contribute to representing a car um in contribute to representing a car um in in that next layer okay so so now we in that next layer okay so so now we in that next layer okay so so now we can't really think there there might can't really think there there might can't really think there there might still be some something I don't know you still be some something I don't know you still be some something I don't know you could call it like a car concept or could call it like a car concept or could call it like a car concept or something but it no longer corresponds something but it no longer corresponds something but it no longer corresponds to a neuron so we need some term for to a neuron so we need some term for to a neuron so we need some term for these kind of neuron-like entities these these kind of neuron-like entities these these kind of neuron-like entities these things that we sort of would have liked things that we sort of would have liked things that we sort of would have liked the neurons to be these idealized the neurons to be these idealized the neurons to be these idealized neurons um the things that are the nice neurons um the things that are the nice neurons um the things that are the nice neurons but also maybe there's more of neurons but also maybe there's more of neurons but also maybe there's more of them somehow hidden and we call those them somehow hidden and we call those them somehow hidden and we call those features and then what are circuits so features and then what are circuits so features and then what are circuits so circuits are these connections of circuits are these connections of circuits are these connections of features right so so when we have the features right so so when we have the features right so so when we have the car detector um and it's connected to a car detector um and it's connected to a car detector um and it's connected to a window detector and a wheel detector and

window detector and a wheel detector and window detector and a wheel detector and it looks for the Wheels below and the it looks for the Wheels below and the it looks for the Wheels below and the windows on top um that's a circuit um so windows on top um that's a circuit um so windows on top um that's a circuit um so circuits are just collections of circuits are just collections of circuits are just collections of features connected by weights um and features connected by weights um and features connected by weights um and they they Implement algorithms so they they they Implement algorithms so they they they Implement algorithms so they tell us you know how is how are features tell us you know how is how are features tell us you know how is how are features used how are they built um how do they used how are they built um how do they used how are they built um how do they connect together so maybe it's it's it's connect together so maybe it's it's it's connect together so maybe it's it's it's worth trying to pin down like what what worth trying to pin down like what what worth trying to pin down like what what really um is the the core hypothesis really um is the the core hypothesis really um is the the core hypothesis here I think the the core hypothesis is here I think the the core hypothesis is here I think the the core hypothesis is something we call the linear something we call the linear something we call the linear representation hypothesis so um if we representation hypothesis so um if we representation hypothesis so um if we think about the car detector you know think about the car detector you know think about the car detector you know the more it fires the more we sort of the more it fires the more we sort of the more it fires the more we sort of think of that as meaning oh the model is think of that as meaning oh the model is think of that as meaning oh the model is more and more confident that um a car more and more confident that um a car more and more confident that um a car was present um or you know if it's some was present um or you know if it's some was present um or you know if it's some combination of neurons that represent a combination of neurons that represent a combination of neurons that represent a car you know the more that combination car you know the more that combination car you know the more that combination fires the more we think the model thinks fires the more we think the model thinks fires the more we think the model thinks there's a car present um this doesn't there's a car present um this doesn't there's a car present um this doesn't have to be the case right like you could have to be the case right like you could have to be the case right like you could imagine something where you have you imagine something where you have you imagine something where you have you know you have this car detector neuron know you have this car detector neuron know you have this car detector neuron and you think ah you know if it fires and you think ah you know if it fires and you think ah you know if it fires like you know between one and two that like you know between one and two that like you know between one and two that means one thing but it means like means one thing but it means like means one thing but it means like totally different if it's between three totally different if it's between three totally different if it's between three and four um that would be a nonlinear and four um that would be a nonlinear and four um that would be a nonlinear representation and principle that you representation and principle that you representation and principle that you know models could do that I think it's know models could do that I think it's know models could do that I think it's it's sort of inefficient for them to do it's sort of inefficient for them to do it's sort of inefficient for them to do if you try to think about how you'd if you try to think about how you'd if you try to think about how you'd Implement computation like that it's Implement computation like that it's Implement computation like that it's it's kind of an annoying thing to do but it's kind of an annoying thing to do but it's kind of an annoying thing to do but in principal models can do that um so uh in principal models can do that um so uh in principal models can do that um so uh one way to think about the features and one way to think about the features and one way to think about the features and and circuits sort of framework for and circuits sort of framework for and circuits sort of framework for thinking about things is that we're thinking about things is that we're thinking about things is that we're thinking about things as being linear thinking about things as being linear thinking about things as being linear we're thinking about there as being um we're thinking about there as being um we're thinking about there as being um that if a if a neuron or a combination that if a if a neuron or a combination that if a if a neuron or a combination of neurons fires more it's sort of that of neurons fires more it's sort of that of neurons fires more it's sort of that means more of the of a particular thing means more of the of a particular thing means more of the of a particular thing being detected and then that gives being detected and then that gives being detected and then that gives weights a very clean interpretation as

weights a very clean interpretation as weights a very clean interpretation as these edges between these these entities these edges between these these entities these edges between these these entities that these features um and that that that these features um and that that that these features um and that that edge then has a has a meaning um so edge then has a has a meaning um so edge then has a has a meaning um so that's that's in some ways the the core that's that's in some ways the the core that's that's in some ways the the core thing um it's it's like um you know we thing um it's it's like um you know we thing um it's it's like um you know we can talk about this sort of outside the can talk about this sort of outside the can talk about this sort of outside the context of ns are you familiar with the context of ns are you familiar with the context of ns are you familiar with the word toac results um so you have like word toac results um so you have like word toac results um so you have like you know King minus man plus woman you know King minus man plus woman you know King minus man plus woman equals Queen well the reason you can do equals Queen well the reason you can do equals Queen well the reason you can do that kind of arithmetic um is because that kind of arithmetic um is because that kind of arithmetic um is because you have a linear representation can you you have a linear representation can you you have a linear representation can you actually explain that representation a actually explain that representation a actually explain that representation a little bit so first off so a feature is little bit so first off so a feature is little bit so first off so a feature is a is a direction of activation you think a is a direction of activation you think a is a direction of activation you think it that way can you do the the the minus it that way can you do the the the minus it that way can you do the the the minus men plus women that that the war Toc men plus women that that the war Toc men plus women that that the war Toc stuff can you explain what that is yeah stuff can you explain what that is yeah stuff can you explain what that is yeah there's this very such a simple clean there's this very such a simple clean there's this very such a simple clean explanation of what we're talking about explanation of what we're talking about explanation of what we're talking about exactly yeah so there's this very famous exactly yeah so there's this very famous exactly yeah so there's this very famous result word toac by um Thomas mikov at result word toac by um Thomas mikov at result word toac by um Thomas mikov at all and there's been tons of follow-up all and there's been tons of follow-up all and there's been tons of follow-up work exploring this so so sometimes we work exploring this so so sometimes we work exploring this so so sometimes we have these we create these word have these we create these word have these we create these word embeddings um where uh we map every word embeddings um where uh we map every word embeddings um where uh we map every word to a vector I mean that in itself by the to a vector I mean that in itself by the to a vector I mean that in itself by the way is is kind of a crazy thing if you way is is kind of a crazy thing if you way is is kind of a crazy thing if you haven't thought about it before right haven't thought about it before right haven't thought about it before right like we we're we're going and and like we we're we're going and and like we we're we're going and and representing we're turning um you know representing we're turning um you know representing we're turning um you know like like if if you just learned about like like if if you just learned about like like if if you just learned about vectors in physics class right uh and vectors in physics class right uh and vectors in physics class right uh and I'm like oh I'm going to actually turn I'm like oh I'm going to actually turn I'm like oh I'm going to actually turn every word uh in the dictionary into a every word uh in the dictionary into a every word uh in the dictionary into a vector that's kind of a crazy idea okay vector that's kind of a crazy idea okay vector that's kind of a crazy idea okay but you could imagine um you could but you could imagine um you could but you could imagine um you could imagine all kinds of ways in which you imagine all kinds of ways in which you imagine all kinds of ways in which you might map words to to might map words to to might map words to to vectors but it it it seems like when we vectors but it it it seems like when we vectors but it it it seems like when we train neural networks um they like to go train neural networks um they like to go train neural networks um they like to go and and map words detectors to such that

and and map words detectors to such that and and map words detectors to such that they're they're they they sort of linear they're they're they they sort of linear they're they're they they sort of linear structure in a particular sense which is structure in a particular sense which is structure in a particular sense which is that directions have meaning so for that directions have meaning so for that directions have meaning so for instance if you there there will be some instance if you there there will be some instance if you there there will be some direction that seems to sort of direction that seems to sort of direction that seems to sort of correspond to gender and male words will correspond to gender and male words will correspond to gender and male words will be you know far in One Direction and be you know far in One Direction and be you know far in One Direction and female words will be in another female words will be in another female words will be in another Direction and the linear representation Direction and the linear representation Direction and the linear representation hypothesis is you you could sort of hypothesis is you you could sort of hypothesis is you you could sort of think of it roughly as saying that think of it roughly as saying that think of it roughly as saying that that's actually kind of the fundamental that's actually kind of the fundamental that's actually kind of the fundamental thing that's going on that that thing that's going on that that thing that's going on that that everything is just different directions everything is just different directions everything is just different directions have meanings and adding different have meanings and adding different have meanings and adding different Direction vectors together can represent Direction vectors together can represent Direction vectors together can represent Concepts and the michelov paper sort of Concepts and the michelov paper sort of Concepts and the michelov paper sort of took that idea seriously and one took that idea seriously and one took that idea seriously and one consequence of it is that you can you consequence of it is that you can you consequence of it is that you can you can do this game of playing sort of can do this game of playing sort of can do this game of playing sort of arithmetic with words so you can do king arithmetic with words so you can do king arithmetic with words so you can do king and you can you know subtract off the and you can you know subtract off the and you can you know subtract off the word man and add the word woman and so word man and add the word woman and so word man and add the word woman and so you're sort of you know going and and you're sort of you know going and and you're sort of you know going and and trying to switch the gender and indeed trying to switch the gender and indeed trying to switch the gender and indeed if you do that the result will sort of if you do that the result will sort of if you do that the result will sort of be close to the word Queen um and you be close to the word Queen um and you be close to the word Queen um and you can you know do other things like you can you know do other things like you can you know do other things like you can do um uh you know Sushi minus Japan can do um uh you know Sushi minus Japan can do um uh you know Sushi minus Japan plus Italy and get pizza or uh different plus Italy and get pizza or uh different plus Italy and get pizza or uh different things like this right um so so this is things like this right um so so this is things like this right um so so this is in some sense the core of the linear in some sense the core of the linear in some sense the core of the linear representation hypothesis you can representation hypothesis you can representation hypothesis you can describe it just as a purely abstract describe it just as a purely abstract describe it just as a purely abstract thing about Vector spaces you can thing about Vector spaces you can thing about Vector spaces you can describe it as a as a statement about um describe it as a as a statement about um describe it as a as a statement about um about the activations of neurons um but about the activations of neurons um but about the activations of neurons um but it's really about this this property of it's really about this this property of it's really about this this property of directions having meaning and in some directions having meaning and in some directions having meaning and in some ways it's even a little subtle than that ways it's even a little subtle than that ways it's even a little subtle than that it's really I think mostly about this it's really I think mostly about this it's really I think mostly about this property of being able to add things property of being able to add things property of being able to add things together um that you can sort of together um that you can sort of together um that you can sort of independently modify um say gender and independently modify um say gender and independently modify um say gender and royalty or royalty or royalty or um you know Cuisine typee or country and

um you know Cuisine typee or country and um you know Cuisine typee or country and and and and the concept of food by by and and and the concept of food by by and and and the concept of food by by adding them do you think the linear adding them do you think the linear adding them do you think the linear hypothesis holds that carries scales so hypothesis holds that carries scales so hypothesis holds that carries scales so so far I think everything I have seen is so far I think everything I have seen is so far I think everything I have seen is consistent with this hypothesis and it consistent with this hypothesis and it consistent with this hypothesis and it doesn't have to be that way right like doesn't have to be that way right like doesn't have to be that way right like like you can write down neural networks like you can write down neural networks like you can write down neural networks where um you write weights such that where um you write weights such that where um you write weights such that they don't have linear representations they don't have linear representations they don't have linear representations where the right way to understand them where the right way to understand them where the right way to understand them is not is not in terms of linear is not is not in terms of linear is not is not in terms of linear representations but I think every representations but I think every representations but I think every natural neural network I've seen um Hess natural neural network I've seen um Hess natural neural network I've seen um Hess property um there's been one paper property um there's been one paper property um there's been one paper recently um that there's been some sort recently um that there's been some sort recently um that there's been some sort of pushing around the edges so I think of pushing around the edges so I think of pushing around the edges so I think there's been some work recently studying there's been some work recently studying there's been some work recently studying multi-dimensional features where rather multi-dimensional features where rather multi-dimensional features where rather than a single Direction it's more like than a single Direction it's more like than a single Direction it's more like um a manifold of directions this to me um a manifold of directions this to me um a manifold of directions this to me still seems like a linear representation still seems like a linear representation still seems like a linear representation um and then there's been some other um and then there's been some other um and then there's been some other papers suggesting that maybe um in in papers suggesting that maybe um in in papers suggesting that maybe um in in very small models you get nonlinear very small models you get nonlinear very small models you get nonlinear representations um I think that the representations um I think that the representations um I think that the jury's still out on that jury's still out on that jury's still out on that um but in I think everything that we've um but in I think everything that we've um but in I think everything that we've seen so far has been consistent with the seen so far has been consistent with the seen so far has been consistent with the linear representation hypothesis and linear representation hypothesis and linear representation hypothesis and that's that's wild it it doesn't have to that's that's wild it it doesn't have to that's that's wild it it doesn't have to be that way um and yet uh I think that be that way um and yet uh I think that be that way um and yet uh I think that there's a lot of evidence that certainly there's a lot of evidence that certainly there's a lot of evidence that certainly at least this is very very widespread at least this is very very widespread at least this is very very widespread and so far the evidence is is consistent and so far the evidence is is consistent and so far the evidence is is consistent with that and I and I I think you know with that and I and I I think you know with that and I and I I think you know one thing you might say is you might say one thing you might say is you might say one thing you might say is you might say well Christopher you know it's that's a well Christopher you know it's that's a well Christopher you know it's that's a lot you know to to go and and sort of um lot you know to to go and and sort of um lot you know to to go and and sort of um to ride on you know if we don't know for to ride on you know if we don't know for to ride on you know if we don't know for sure this is true and you're sort of you sure this is true and you're sort of you sure this is true and you're sort of you know you're investigating all not works know you're investigating all not works know you're investigating all not works as though it is true you know isn't that as though it is true you know isn't that as though it is true you know isn't that um isn't that dangerous well you know um isn't that dangerous well you know um isn't that dangerous well you know but I I think actually there's a there's but I I think actually there's a there's but I I think actually there's a there's a virtue in taking hypotheses seriously

a virtue in taking hypotheses seriously a virtue in taking hypotheses seriously and pushing them as far as they can go and pushing them as far as they can go and pushing them as far as they can go um so it might be that someday we um so it might be that someday we um so it might be that someday we discover something that is inconsistent discover something that is inconsistent discover something that is inconsistent with linear representation hypothesis with linear representation hypothesis with linear representation hypothesis but science is full of hypothesis and but science is full of hypothesis and but science is full of hypothesis and theories that were wrong um and we theories that were wrong um and we theories that were wrong um and we learned a lot by sort of working under learned a lot by sort of working under learned a lot by sort of working under under them as a sort of an assumption um under them as a sort of an assumption um under them as a sort of an assumption um and and then going and pushing them as and and then going and pushing them as and and then going and pushing them as far as we can I guess I guess this is far as we can I guess I guess this is far as we can I guess I guess this is sort of the heart of of what would sort of the heart of of what would sort of the heart of of what would call normal normal science um um I don't call normal normal science um um I don't call normal normal science um um I don't know if you want we can talk a lot about know if you want we can talk a lot about know if you want we can talk a lot about about uh philosophy of science and uh about uh philosophy of science and uh about uh philosophy of science and uh that leads to the paradigm shift so yeah that leads to the paradigm shift so yeah that leads to the paradigm shift so yeah I love it taking the hypothesis I love it taking the hypothesis I love it taking the hypothesis seriously and take it to a natural seriously and take it to a natural seriously and take it to a natural natural conclusion yeah same with the natural conclusion yeah same with the natural conclusion yeah same with the scaling hypothesis same exactly exactly scaling hypothesis same exactly exactly scaling hypothesis same exactly exactly and I love it one of my colleagues Tom and I love it one of my colleagues Tom and I love it one of my colleagues Tom henigan who is a former physicist um henigan who is a former physicist um henigan who is a former physicist um like made this really nice analogy to me like made this really nice analogy to me like made this really nice analogy to me of um uh caloric Theory where you know of um uh caloric Theory where you know of um uh caloric Theory where you know once upon a time we thought that heat once upon a time we thought that heat once upon a time we thought that heat was actually you know this thing called was actually you know this thing called was actually you know this thing called caloric and like the reason you know hot caloric and like the reason you know hot caloric and like the reason you know hot objects you know would would warm up objects you know would would warm up objects you know would would warm up cool objects is like the caloric is cool objects is like the caloric is cool objects is like the caloric is flowing through them um and like you flowing through them um and like you flowing through them um and like you know because we're so used to thinking know because we're so used to thinking know because we're so used to thinking about about heat you know in terms of about about heat you know in terms of about about heat you know in terms of the modern modern Theory you know that the modern modern Theory you know that the modern modern Theory you know that seems kind of silly but it's actually seems kind of silly but it's actually seems kind of silly but it's actually very hard to construct uh an experiment very hard to construct uh an experiment very hard to construct uh an experiment that that sort of disproves the um that that sort of disproves the um that that sort of disproves the um chloric hypothesis um and you know you chloric hypothesis um and you know you chloric hypothesis um and you know you can actually do a lot of really useful can actually do a lot of really useful can actually do a lot of really useful work believing in chloric for example it work believing in chloric for example it work believing in chloric for example it turns out that the original combustion turns out that the original combustion turns out that the original combustion engines were developed by people who engines were developed by people who engines were developed by people who believe in the caloric Theory so I think believe in the caloric Theory so I think believe in the caloric Theory so I think this a virtue in taking hypotheses this a virtue in taking hypotheses this a virtue in taking hypotheses seriously even when they might be wrong

seriously even when they might be wrong seriously even when they might be wrong yeah yeah there's a deep philosophical yeah yeah there's a deep philosophical yeah yeah there's a deep philosophical truth to that that's kind of kind of how truth to that that's kind of kind of how truth to that that's kind of kind of how I feel about space travel like I feel about space travel like I feel about space travel like colonizing Mars there's a lot of people colonizing Mars there's a lot of people colonizing Mars there's a lot of people that criticize that I think if you just that criticize that I think if you just that criticize that I think if you just assume we have to colonize Mars in order assume we have to colonize Mars in order assume we have to colonize Mars in order to have a backup for human civilization to have a backup for human civilization to have a backup for human civilization even if that's not true that's going to even if that's not true that's going to even if that's not true that's going to produce some interesting interesting produce some interesting interesting produce some interesting interesting engineering and even scientific engineering and even scientific engineering and even scientific breakthroughs I think yeah well and breakthroughs I think yeah well and breakthroughs I think yeah well and actually this is another thing that I actually this is another thing that I actually this is another thing that I think is really interesting so um you think is really interesting so um you think is really interesting so um you know there a way in which I think it can know there a way in which I think it can know there a way in which I think it can be really useful for society to have be really useful for society to have be really useful for society to have people um almost irrationally dedicated people um almost irrationally dedicated people um almost irrationally dedicated to investigating particular hypothesis to investigating particular hypothesis to investigating particular hypothesis um because uh well it it takes a lot to um because uh well it it takes a lot to um because uh well it it takes a lot to sort of maintain scientific morale and sort of maintain scientific morale and sort of maintain scientific morale and really push on something when you know really push on something when you know really push on something when you know most most SCI scientific hypotheses end most most SCI scientific hypotheses end most most SCI scientific hypotheses end up being wrong you know a lot of a lot up being wrong you know a lot of a lot up being wrong you know a lot of a lot of science doesn't doesn't work out um of science doesn't doesn't work out um of science doesn't doesn't work out um and but and yet it's you know it's very and but and yet it's you know it's very and but and yet it's you know it's very it's very useful to go do you know um it's very useful to go do you know um it's very useful to go do you know um there's a there's a joke about Jeff there's a there's a joke about Jeff there's a there's a joke about Jeff Hinton um which is that uh Jeff Hinton Hinton um which is that uh Jeff Hinton Hinton um which is that uh Jeff Hinton has discovered how the brain works every has discovered how the brain works every has discovered how the brain works every year for the last 50 years yeah um but year for the last 50 years yeah um but year for the last 50 years yeah um but you know I I say that with like you know you know I I say that with like you know you know I I say that with like you know the you know with really deep respect the you know with really deep respect the you know with really deep respect because uh in fact that's actually you because uh in fact that's actually you because uh in fact that's actually you know that that led to him doing some know that that led to him doing some know that that led to him doing some some really great work yeah he won the some really great work yeah he won the some really great work yeah he won the Noel prize Now Who's Laughing Now Noel prize Now Who's Laughing Now Noel prize Now Who's Laughing Now exactly exactly exactly um yeah I think exactly exactly exactly um yeah I think exactly exactly exactly um yeah I think one want to be able to pop up and sort one want to be able to pop up and sort one want to be able to pop up and sort of recognize the the appropriate level of recognize the the appropriate level of recognize the the appropriate level of confidence but I think there's also a of confidence but I think there's also a of confidence but I think there's also a lot of value and just being like you lot of value and just being like you lot of value and just being like you know I'm going to essentially assume I'm know I'm going to essentially assume I'm know I'm going to essentially assume I'm going to condition on this problem being going to condition on this problem being going to condition on this problem being possible or this being broadly the right

possible or this being broadly the right possible or this being broadly the right approach and I'm just going to go and approach and I'm just going to go and approach and I'm just going to go and assume that for a while and go and work assume that for a while and go and work assume that for a while and go and work within that um and push really hard on within that um and push really hard on within that um and push really hard on it um and you know if Society has lots it um and you know if Society has lots it um and you know if Society has lots of people doing doing that for different of people doing doing that for different of people doing doing that for different things um that's actually really useful things um that's actually really useful things um that's actually really useful in terms of going and uh getting in terms of going and uh getting in terms of going and uh getting to getting you know either really really to getting you know either really really to getting you know either really really ruling things out right we can be like ruling things out right we can be like ruling things out right we can be like well you know that didn't work we know well you know that didn't work we know well you know that didn't work we know that somebody tried hard um or going in that somebody tried hard um or going in that somebody tried hard um or going in and getting to something that that does and getting to something that that does and getting to something that that does teach us something about the world so teach us something about the world so teach us something about the world so another interesting hypothesis is the another interesting hypothesis is the another interesting hypothesis is the superposition hypothesis can you superposition hypothesis can you superposition hypothesis can you describe what superos is yeah so earlier describe what superos is yeah so earlier describe what superos is yeah so earlier we were talking about word toac right we were talking about word toac right we were talking about word toac right and we were talking about how you know and we were talking about how you know and we were talking about how you know maybe you have One Direction that maybe you have One Direction that maybe you have One Direction that corresponds to gender and maybe another corresponds to gender and maybe another corresponds to gender and maybe another that corresponds to royalty and another that corresponds to royalty and another that corresponds to royalty and another one that corresponds to Italy and one that corresponds to Italy and one that corresponds to Italy and another one that corresponds to you know another one that corresponds to you know another one that corresponds to you know food and and all these things well you food and and all these things well you food and and all these things well you know um often times maybe these these uh know um often times maybe these these uh know um often times maybe these these uh these Ward embeddings they might be 500 these Ward embeddings they might be 500 these Ward embeddings they might be 500 dimensions a thousand dimensions and so dimensions a thousand dimensions and so dimensions a thousand dimensions and so if you believed that all of those if you believed that all of those if you believed that all of those directions were directions were directions were orthogonal um then you could only have orthogonal um then you could only have orthogonal um then you could only have you know 500 Concepts and you know I I you know 500 Concepts and you know I I you know 500 Concepts and you know I I love pizza um but like if I was going to love pizza um but like if I was going to love pizza um but like if I was going to go and like give the like 500 most go and like give the like 500 most go and like give the like 500 most important Concepts in um you know the important Concepts in um you know the important Concepts in um you know the English language probably Italy wouldn't English language probably Italy wouldn't English language probably Italy wouldn't be it's not obvious at least that Italy be it's not obvious at least that Italy be it's not obvious at least that Italy would be one of them right because you would be one of them right because you would be one of them right because you you have to have things like plural and you have to have things like plural and you have to have things like plural and singular and U uh verb and noun and singular and U uh verb and noun and singular and U uh verb and noun and adjective and you know um there's a lot adjective and you know um there's a lot adjective and you know um there's a lot of things we have to get to before we of things we have to get to before we of things we have to get to before we get to get to Italy um uh and Japan and get to get to Italy um uh and Japan and get to get to Italy um uh and Japan and you know there's a lot of countries in you know there's a lot of countries in you know there's a lot of countries in the world um and so how might it be that

the world um and so how might it be that the world um and so how might it be that models could you know simultaneously models could you know simultaneously models could you know simultaneously have the linear representation have the linear representation have the linear representation hypothesis be true and also represent hypothesis be true and also represent hypothesis be true and also represent more things than they have directions so more things than they have directions so more things than they have directions so so what does that mean well okay so if so what does that mean well okay so if so what does that mean well okay so if if if linear representation hypothesis if if linear representation hypothesis if if linear representation hypothesis is true something interesting has to be is true something interesting has to be is true something interesting has to be going on now I'll I'll tell you one more going on now I'll I'll tell you one more going on now I'll I'll tell you one more interesting thing before we we go and we interesting thing before we we go and we interesting thing before we we go and we do that which is um you know earlier we do that which is um you know earlier we do that which is um you know earlier we were talking about all these polymatic were talking about all these polymatic were talking about all these polymatic neurons right um these neurons that you neurons right um these neurons that you neurons right um these neurons that you know when we're looking at Inception V1 know when we're looking at Inception V1 know when we're looking at Inception V1 there's these nice neurons that like the there's these nice neurons that like the there's these nice neurons that like the car detector and the curve detector and car detector and the curve detector and car detector and the curve detector and so on that respond to lots of you know so on that respond to lots of you know so on that respond to lots of you know to very coherent things but it's lots of to very coherent things but it's lots of to very coherent things but it's lots of neurons that respond to a bunch of neurons that respond to a bunch of neurons that respond to a bunch of unrelated things that's that's also an unrelated things that's that's also an unrelated things that's that's also an interesting phenomenon um and it turns interesting phenomenon um and it turns interesting phenomenon um and it turns out as well that even these neurons that out as well that even these neurons that out as well that even these neurons that are really really clean if you look at are really really clean if you look at are really really clean if you look at the weak activations right so if you the weak activations right so if you the weak activations right so if you look at like you know the activation look at like you know the activation look at like you know the activation where it's like activating 5% of of the where it's like activating 5% of of the where it's like activating 5% of of the the you know of the maximum activation the you know of the maximum activation the you know of the maximum activation it's really not the core thing that it's it's really not the core thing that it's it's really not the core thing that it's expecting right so if you look at a a expecting right so if you look at a a expecting right so if you look at a a curve detector for instance and you look curve detector for instance and you look curve detector for instance and you look at the places where it's 5% active you at the places where it's 5% active you at the places where it's 5% active you know you could interpret it just as know you could interpret it just as know you could interpret it just as noise or it could be that it's that it's noise or it could be that it's that it's noise or it could be that it's that it's doing something else there okay so so doing something else there okay so so doing something else there okay so so how could that be how could that be how could that be well there's this amazing thing in well there's this amazing thing in well there's this amazing thing in mathematics um called compressed sensing mathematics um called compressed sensing mathematics um called compressed sensing and it's it's actually this this very and it's it's actually this this very and it's it's actually this this very surprising fact where if you have a high surprising fact where if you have a high surprising fact where if you have a high dimensional space and you project it dimensional space and you project it dimensional space and you project it into a low dimensional space ordinarily into a low dimensional space ordinarily into a low dimensional space ordinarily you can't go and sort of unprojected and you can't go and sort of unprojected and you can't go and sort of unprojected and get back your high dimensional Vector get back your high dimensional Vector get back your high dimensional Vector right you threw information away this is right you threw information away this is right you threw information away this is like you know you can't you can't invert like you know you can't you can't invert like you know you can't you can't invert a rectangular Matrix um you can only

a rectangular Matrix um you can only a rectangular Matrix um you can only invert Square invert Square invert Square matrices um but it turns out that that's matrices um but it turns out that that's matrices um but it turns out that that's actually not quite true if I tell you actually not quite true if I tell you actually not quite true if I tell you that the high dimensional Vector was that the high dimensional Vector was that the high dimensional Vector was sparse so it's mostly zeros then it sparse so it's mostly zeros then it sparse so it's mostly zeros then it turns out that you can often go and find turns out that you can often go and find turns out that you can often go and find back um the uh the high dimensional back um the uh the high dimensional back um the uh the high dimensional Vector with with very high probability Vector with with very high probability Vector with with very high probability um so that's a surprising fact right it um so that's a surprising fact right it um so that's a surprising fact right it says that you know you can um you can says that you know you can um you can says that you know you can um you can you can have this High dimensional you can have this High dimensional you can have this High dimensional Vector space and as long as things are Vector space and as long as things are Vector space and as long as things are sparse um you can project it down you sparse um you can project it down you sparse um you can project it down you can have a lower dimensional projection can have a lower dimensional projection can have a lower dimensional projection of it and that works so the super of it and that works so the super of it and that works so the super hypothesis is saying that that's what's hypothesis is saying that that's what's hypothesis is saying that that's what's going on in neural networks that's for going on in neural networks that's for going on in neural networks that's for instance that's what's going on in wart instance that's what's going on in wart instance that's what's going on in wart edings the wart embeddings are able to edings the wart embeddings are able to edings the wart embeddings are able to simultaneously have directions be the simultaneously have directions be the simultaneously have directions be the meaningful thing and by exploiting the meaningful thing and by exploiting the meaningful thing and by exploiting the fact that they're they're operating on a fact that they're they're operating on a fact that they're they're operating on a fairly High dimensional space they're fairly High dimensional space they're fairly High dimensional space they're actually and and the fact that these actually and and the fact that these actually and and the fact that these concepts are right like you know you concepts are right like you know you concepts are right like you know you usually aren't talking about Japan and usually aren't talking about Japan and usually aren't talking about Japan and Italy at the same time um you know most Italy at the same time um you know most Italy at the same time um you know most of the most of those Concepts you know of the most of those Concepts you know of the most of those Concepts you know in most sentences Japan and Italy are in most sentences Japan and Italy are in most sentences Japan and Italy are both zero they're not present at all um both zero they're not present at all um both zero they're not present at all um and if that's true um then you can go and if that's true um then you can go and if that's true um then you can go and have it be the case that um that you and have it be the case that um that you and have it be the case that um that you can you can have many more of these sort can you can have many more of these sort can you can have many more of these sort of directions that are meaningful these of directions that are meaningful these of directions that are meaningful these features than you have dimensions and features than you have dimensions and features than you have dimensions and some of when we're talking about neurons some of when we're talking about neurons some of when we're talking about neurons you can have many more Concepts than you you can have many more Concepts than you you can have many more Concepts than you have have neurons so that's the at a have have neurons so that's the at a have have neurons so that's the at a high level super hypothesis now it has high level super hypothesis now it has high level super hypothesis now it has this even Wilder implication which is um this even Wilder implication which is um this even Wilder implication which is um to go and say that uh neural networks to go and say that uh neural networks to go and say that uh neural networks are it may not just be the case that the

are it may not just be the case that the are it may not just be the case that the the representations are like of this but the representations are like of this but the representations are like of this but the the computation may also be like the the computation may also be like the the computation may also be like this you know the connections between this you know the connections between this you know the connections between all of them and so in in some sense all of them and so in in some sense all of them and so in in some sense neural networks may be shadows of much neural networks may be shadows of much neural networks may be shadows of much larger sparer neural networks and what larger sparer neural networks and what larger sparer neural networks and what we see are these we see are these we see are these projections um and the super you the projections um and the super you the projections um and the super you the strongest version of the super strongest version of the super strongest version of the super hypothesis would be to take that really hypothesis would be to take that really hypothesis would be to take that really seriously and sort of say you know there seriously and sort of say you know there seriously and sort of say you know there there actually is in some sense this there actually is in some sense this there actually is in some sense this this upstairs model this you know um this upstairs model this you know um this upstairs model this you know um where where the neurons are really where where the neurons are really where where the neurons are really sparse and all interpal and there's you sparse and all interpal and there's you sparse and all interpal and there's you know the weights between them are these know the weights between them are these know the weights between them are these really sparse circuits and that's what really sparse circuits and that's what really sparse circuits and that's what we're we're we're studying um and uh the thing that we're studying um and uh the thing that we're studying um and uh the thing that we're observing is the shadow of it and we observing is the shadow of it and we observing is the shadow of it and we need to find the original object and uh need to find the original object and uh need to find the original object and uh the process of learning is trying to the process of learning is trying to the process of learning is trying to construct a compression of the upstairs construct a compression of the upstairs construct a compression of the upstairs model that doesn't lose too much model that doesn't lose too much model that doesn't lose too much information in the projection yeah information in the projection yeah information in the projection yeah finding how to fit it efficiently or finding how to fit it efficiently or finding how to fit it efficiently or something like this um that grent is something like this um that grent is something like this um that grent is doing this in fact so this sort of says doing this in fact so this sort of says doing this in fact so this sort of says that gradient descent you know could it that gradient descent you know could it that gradient descent you know could it could just represent a dense neural could just represent a dense neural could just represent a dense neural network but it sort of says that network but it sort of says that network but it sort of says that gradient descent is pleasantly searching gradient descent is pleasantly searching gradient descent is pleasantly searching over the space of extremely sparse over the space of extremely sparse over the space of extremely sparse models that could be projected into this models that could be projected into this models that could be projected into this low dimensional space and this large low dimensional space and this large low dimensional space and this large body of work of of people going and body of work of of people going and body of work of of people going and trying to study sparse neural networks trying to study sparse neural networks trying to study sparse neural networks right where you go and you have you right where you go and you have you right where you go and you have you could design neural networks right where could design neural networks right where could design neural networks right where where the edges are sparse and the where the edges are sparse and the where the edges are sparse and the activations are sparse and you know my activations are sparse and you know my activations are sparse and you know my sense is that work has gener sense is that work has gener sense is that work has gener it feels very principled right it makes it feels very principled right it makes it feels very principled right it makes so much sense and yet that that work so much sense and yet that that work so much sense and yet that that work hasn't really panned out that well as my hasn't really panned out that well as my hasn't really panned out that well as my impression broadly and I think that a a impression broadly and I think that a a impression broadly and I think that a a potential answer for that is that potential answer for that is that potential answer for that is that actually the neural network is already

actually the neural network is already actually the neural network is already sparse in some sense grading descent was sparse in some sense grading descent was sparse in some sense grading descent was the whole time gradi you were trying to the whole time gradi you were trying to the whole time gradi you were trying to go and do this gradiant descent was go and do this gradiant descent was go and do this gradiant descent was actually in the behind the scenes going actually in the behind the scenes going actually in the behind the scenes going and searching more efficiently than you and searching more efficiently than you and searching more efficiently than you could through the space of sparse models could through the space of sparse models could through the space of sparse models and going in learning whatever sparse and going in learning whatever sparse and going in learning whatever sparse model was most efficient and then model was most efficient and then model was most efficient and then figuring out how to fold it down nicely figuring out how to fold it down nicely figuring out how to fold it down nicely to go and run conven on your GPU which to go and run conven on your GPU which to go and run conven on your GPU which does you know nice dense Matrix does you know nice dense Matrix does you know nice dense Matrix multiplies um and that you just can't multiplies um and that you just can't multiplies um and that you just can't beat that how many Concepts do you think beat that how many Concepts do you think beat that how many Concepts do you think can be shoved in into a neural network can be shoved in into a neural network can be shoved in into a neural network depends on how sparse they are so there depends on how sparse they are so there depends on how sparse they are so there there's probably an upper bound from the there's probably an upper bound from the there's probably an upper bound from the number of parameters right because you number of parameters right because you number of parameters right because you have to have you still have to have you have to have you still have to have you have to have you still have to have you know print weights that go and connect know print weights that go and connect know print weights that go and connect them together um so that's that's one them together um so that's that's one them together um so that's that's one upper bound there are in fact all these upper bound there are in fact all these upper bound there are in fact all these lovely results from compressed sensing lovely results from compressed sensing lovely results from compressed sensing and the Johnson Linton stess Lemma and and the Johnson Linton stess Lemma and and the Johnson Linton stess Lemma and and things like this um that they they and things like this um that they they and things like this um that they they basically tell you that if you have a basically tell you that if you have a basically tell you that if you have a vector space and you want to have almost vector space and you want to have almost vector space and you want to have almost orthogonal vectors which is sort of orthogonal vectors which is sort of orthogonal vectors which is sort of probably the thing that you want here probably the thing that you want here probably the thing that you want here right so you you're going to say well right so you you're going to say well right so you you're going to say well you know I'm going to give up on having you know I'm going to give up on having you know I'm going to give up on having my my Concepts my features be strictly my my Concepts my features be strictly my my Concepts my features be strictly orthogonal but I'd like them to not orthogonal but I'd like them to not orthogonal but I'd like them to not interfere that much I'm going to have to interfere that much I'm going to have to interfere that much I'm going to have to ask them to be almost orthogonal um then ask them to be almost orthogonal um then ask them to be almost orthogonal um then this would say that it's actually you this would say that it's actually you this would say that it's actually you know for once you set a threshold for know for once you set a threshold for know for once you set a threshold for for what you're what you're willing to for what you're what you're willing to for what you're what you're willing to accept in terms of how how much coine accept in terms of how how much coine accept in terms of how how much coine similarity there is that's actually similarity there is that's actually similarity there is that's actually exponential in the number of neurons exponential in the number of neurons exponential in the number of neurons that you have so at some point that's that you have so at some point that's that you have so at some point that's not going to even be the the limiting not going to even be the the limiting not going to even be the the limiting factor um but um there some beautiful factor um but um there some beautiful factor um but um there some beautiful results there and in fact it's probably results there and in fact it's probably results there and in fact it's probably even better than that in some sense even better than that in some sense even better than that in some sense because that's sort of is for saying because that's sort of is for saying because that's sort of is for saying that you know any random set of features that you know any random set of features that you know any random set of features could be active but in fact the features could be active but in fact the features could be active but in fact the features have sort of a correlational structure have sort of a correlational structure have sort of a correlational structure where some features you know are more

where some features you know are more where some features you know are more likely to co-occur and other ones are likely to co-occur and other ones are likely to co-occur and other ones are less likely to co-occur and so neural less likely to co-occur and so neural less likely to co-occur and so neural networks my guess would be can do do networks my guess would be can do do networks my guess would be can do do very well in terms of going and uh very well in terms of going and uh very well in terms of going and uh packing things in such to to the point packing things in such to to the point packing things in such to to the point that's probably probably not the that's probably probably not the that's probably probably not the limiting factor how does the problem of limiting factor how does the problem of limiting factor how does the problem of polys semanticity enter the picture here polys semanticity enter the picture here polys semanticity enter the picture here poly semanticity is this phenomenon we poly semanticity is this phenomenon we poly semanticity is this phenomenon we observe where we look at many neurons observe where we look at many neurons observe where we look at many neurons and the neuron doesn't just sort of and the neuron doesn't just sort of and the neuron doesn't just sort of represent one one concept it's not it's represent one one concept it's not it's represent one one concept it's not it's not a clean feature it responds to a not a clean feature it responds to a not a clean feature it responds to a bunch of unrelated things and um bunch of unrelated things and um bunch of unrelated things and um supersition is you can think of as as supersition is you can think of as as supersition is you can think of as as being a hypothesis that explains the being a hypothesis that explains the being a hypothesis that explains the observation of polys semanticity um so observation of polys semanticity um so observation of polys semanticity um so poly semanticity is this observe poly semanticity is this observe poly semanticity is this observe phenomenon and super is is a hypothesis phenomenon and super is is a hypothesis phenomenon and super is is a hypothesis that um would explain it along with with that um would explain it along with with that um would explain it along with with some other so that makes Mech turb more some other so that makes Mech turb more some other so that makes Mech turb more difficult right so if you if you're difficult right so if you if you're difficult right so if you if you're trying to understand things in terms of trying to understand things in terms of trying to understand things in terms of individual neurons and you have individual neurons and you have individual neurons and you have polymatic neurons you're in an awful lot polymatic neurons you're in an awful lot polymatic neurons you're in an awful lot of trouble right I mean the easiest of trouble right I mean the easiest of trouble right I mean the easiest answer is like okay well you know you're answer is like okay well you know you're answer is like okay well you know you're looking at the neurons you're trying to looking at the neurons you're trying to looking at the neurons you're trying to understand them this one responds to a understand them this one responds to a understand them this one responds to a lot of things it doesn't have a nice lot of things it doesn't have a nice lot of things it doesn't have a nice meaning okay we're you that's that's meaning okay we're you that's that's meaning okay we're you that's that's that's bad um another thing you could that's bad um another thing you could that's bad um another thing you could ask is you know ultimately we want to ask is you know ultimately we want to ask is you know ultimately we want to understand the weights and if you have understand the weights and if you have understand the weights and if you have two polymatic neurons and you know each two polymatic neurons and you know each two polymatic neurons and you know each one responds to three things and then one responds to three things and then one responds to three things and then you know the other neuron responds to you know the other neuron responds to you know the other neuron responds to three things and you have weight between three things and you have weight between three things and you have weight between them you know what does that mean does them you know what does that mean does them you know what does that mean does it mean that like all three you know it mean that like all three you know it mean that like all three you know like there's these nine you know nine like there's these nine you know nine like there's these nine you know nine interactions going on it's a very weird interactions going on it's a very weird interactions going on it's a very weird thing but there's also a deeper reason thing but there's also a deeper reason thing but there's also a deeper reason which is related to the fact that neural which is related to the fact that neural which is related to the fact that neural networks operate on really high networks operate on really high networks operate on really high dimensional spaces so I said that our dimensional spaces so I said that our dimensional spaces so I said that our goal was you know to understand neural goal was you know to understand neural goal was you know to understand neural networks and understand the mechanisms networks and understand the mechanisms networks and understand the mechanisms and one thing you might say is like well

and one thing you might say is like well and one thing you might say is like well why not it's just a mathematical why not it's just a mathematical why not it's just a mathematical function why not just look at it right function why not just look at it right function why not just look at it right like um you know one of the earliest like um you know one of the earliest like um you know one of the earliest projects I did studied these these projects I did studied these these projects I did studied these these neural networks that mapped two- neural networks that mapped two- neural networks that mapped two- dimensional spaces to two- dimensional dimensional spaces to two- dimensional dimensional spaces to two- dimensional spaces and you can sort of interpret spaces and you can sort of interpret spaces and you can sort of interpret them in this beautiful way is like them in this beautiful way is like them in this beautiful way is like bending manifolds mhm um why can't we do bending manifolds mhm um why can't we do bending manifolds mhm um why can't we do that well you know as you have have a that well you know as you have have a that well you know as you have have a higher dimensional space um the volume higher dimensional space um the volume higher dimensional space um the volume of that space in some senses is of that space in some senses is of that space in some senses is exponential in the number of inputs you exponential in the number of inputs you exponential in the number of inputs you have and so you can't just go in have and so you can't just go in have and so you can't just go in visualize it so we somehow need to break visualize it so we somehow need to break visualize it so we somehow need to break that apart we need to somehow break that that apart we need to somehow break that that apart we need to somehow break that exponential space into a bunch of things exponential space into a bunch of things exponential space into a bunch of things that we you know some non-exponential that we you know some non-exponential that we you know some non-exponential number of things that we can reason number of things that we can reason number of things that we can reason about independently and the independence about independently and the independence about independently and the independence is crucial because it's the Independence is crucial because it's the Independence is crucial because it's the Independence that allows you to not have to think that allows you to not have to think that allows you to not have to think about you know all the exponential about you know all the exponential about you know all the exponential combinations of things and combinations of things and combinations of things and things being monomatic things only things being monomatic things only things being monomatic things only having one meaning things having a having one meaning things having a having one meaning things having a meaning that isn't is the key thing that meaning that isn't is the key thing that meaning that isn't is the key thing that allows you to think about them allows you to think about them allows you to think about them independently and so I think that's that independently and so I think that's that independently and so I think that's that if you want the deepest reason why we if you want the deepest reason why we if you want the deepest reason why we want to have um interpal monatic want to have um interpal monatic want to have um interpal monatic features I think that's really the the features I think that's really the the features I think that's really the the Deep reason and so the goal here as your Deep reason and so the goal here as your Deep reason and so the goal here as your recent work has been aiming at is how do recent work has been aiming at is how do recent work has been aiming at is how do we extract the mod semantic features we extract the mod semantic features we extract the mod semantic features from a neural net that has politic from a neural net that has politic from a neural net that has politic features and all this this mess yes we features and all this this mess yes we features and all this this mess yes we have the have we observe these polyur have the have we observe these polyur have the have we observe these polyur and we hypothesize that's what's going and we hypothesize that's what's going and we hypothesize that's what's going what's going on at superos and if what's going on at superos and if what's going on at superos and if superos is what's going on there there's superos is what's going on there there's superos is what's going on there there's actually a sort of wellestablished actually a sort of wellestablished actually a sort of wellestablished technique that is sort of the principled technique that is sort of the principled technique that is sort of the principled thing to do which is dictionary learning thing to do which is dictionary learning thing to do which is dictionary learning and um it turns out if you do dictionary

and um it turns out if you do dictionary and um it turns out if you do dictionary learning in particular if you do sort of learning in particular if you do sort of learning in particular if you do sort of a nice efficient way that in some in a nice efficient way that in some in a nice efficient way that in some in some sense sort of nicely regularizes it some sense sort of nicely regularizes it some sense sort of nicely regularizes it well as well called a sparse Auto well as well called a sparse Auto well as well called a sparse Auto encoder if you train a sparse Auto encoder if you train a sparse Auto encoder if you train a sparse Auto encoder these beautiful interpal encoder these beautiful interpal encoder these beautiful interpal features start to just fall out where features start to just fall out where features start to just fall out where there weren't any beforehand and so there weren't any beforehand and so there weren't any beforehand and so that's notot of thing that you would that's notot of thing that you would that's notot of thing that you would necessarily predict right but it turns necessarily predict right but it turns necessarily predict right but it turns out that that works very very well you out that that works very very well you out that that works very very well you know to me that seems like you know some know to me that seems like you know some know to me that seems like you know some non-trivial validation of linear non-trivial validation of linear non-trivial validation of linear representations and supersession so with representations and supersession so with representations and supersession so with dictionary learning you're not looking dictionary learning you're not looking dictionary learning you're not looking for particular kind of categories you for particular kind of categories you for particular kind of categories you don't know what they don't know what they don't know what they arege and this gets back to our earlier arege and this gets back to our earlier arege and this gets back to our earlier point right when we're not making point right when we're not making point right when we're not making assumptions grading descent is smarter assumptions grading descent is smarter assumptions grading descent is smarter than us so we're not making assumptions than us so we're not making assumptions than us so we're not making assumptions about what's there um I mean one about what's there um I mean one about what's there um I mean one certainly could do that right one could certainly could do that right one could certainly could do that right one could assume that there's a PHP feature and go assume that there's a PHP feature and go assume that there's a PHP feature and go and search for it but we're not doing and search for it but we're not doing and search for it but we're not doing that we're saying we don't know what's that we're saying we don't know what's that we're saying we don't know what's going to be there instead we're just going to be there instead we're just going to be there instead we're just going to go and let um the sparse Auto going to go and let um the sparse Auto going to go and let um the sparse Auto encoder discover the things that are encoder discover the things that are encoder discover the things that are there so can you uh talk to the to monos there so can you uh talk to the to monos there so can you uh talk to the to monos semanticity paper from October last year semanticity paper from October last year semanticity paper from October last year that had a lot of like nice breakthrough that had a lot of like nice breakthrough that had a lot of like nice breakthrough results that's very kind of you to results that's very kind of you to results that's very kind of you to describe it that way um yeah I mean this describe it that way um yeah I mean this describe it that way um yeah I mean this was um uh our first real success using was um uh our first real success using was um uh our first real success using sparse Auto encoders so we took a one sparse Auto encoders so we took a one sparse Auto encoders so we took a one layer model um and it turns out if you layer model um and it turns out if you layer model um and it turns out if you go and you you know do dictionary go and you you know do dictionary go and you you know do dictionary learning on it you find all these really learning on it you find all these really learning on it you find all these really nice interpal features so you know the nice interpal features so you know the nice interpal features so you know the Arabic feature the Hebrew feature um the Arabic feature the Hebrew feature um the Arabic feature the Hebrew feature um the Bas 64 feature those were were some some Bas 64 feature those were were some some Bas 64 feature those were were some some examples that we studied in a lot of examples that we studied in a lot of examples that we studied in a lot of depth and really showed that they were depth and really showed that they were depth and really showed that they were um what we thought they were it turns if um what we thought they were it turns if um what we thought they were it turns if you train a model twice as well and you train a model twice as well and you train a model twice as well and train two different models and and do train two different models and and do train two different models and and do dictionary learning you find find dictionary learning you find find dictionary learning you find find analogous features in both of them so

analogous features in both of them so analogous features in both of them so that's fun um you find all kinds of of that's fun um you find all kinds of of that's fun um you find all kinds of of different features so that was really different features so that was really different features so that was really just showing um that um that this works just showing um that um that this works just showing um that um that this works and um you know I should mention that and um you know I should mention that and um you know I should mention that there was this cunning home at all um there was this cunning home at all um there was this cunning home at all um that had very similar results around the that had very similar results around the that had very similar results around the same time there's something fun about same time there's something fun about same time there's something fun about being doing these kinds of small scale being doing these kinds of small scale being doing these kinds of small scale experiments and finding that it's experiments and finding that it's experiments and finding that it's actually working yeah well and there's actually working yeah well and there's actually working yeah well and there's and there's so much structure here like and there's so much structure here like and there's so much structure here like you you know so maybe maybe stepping you you know so maybe maybe stepping you you know so maybe maybe stepping back for a while um I thought that maybe back for a while um I thought that maybe back for a while um I thought that maybe all this mechanistic can really work um all this mechanistic can really work um all this mechanistic can really work um the end result was going to be that I the end result was going to be that I the end result was going to be that I would have an explanation for why it was would have an explanation for why it was would have an explanation for why it was sort of you know very hard and not going sort of you know very hard and not going sort of you know very hard and not going to be tractable um you know we'd be like to be tractable um you know we'd be like to be tractable um you know we'd be like well there's this problem with well there's this problem with well there's this problem with supersession and it turns that super supersession and it turns that super supersession and it turns that super session is really hard um and we're kind session is really hard um and we're kind session is really hard um and we're kind of screwed but that's not what happened of screwed but that's not what happened of screwed but that's not what happened in fact a very natural Le technique just in fact a very natural Le technique just in fact a very natural Le technique just works and so then that's actually a very works and so then that's actually a very works and so then that's actually a very good situation you know I think um this good situation you know I think um this good situation you know I think um this is a sort of hard research problem and is a sort of hard research problem and is a sort of hard research problem and it's got a lot of research risk and you it's got a lot of research risk and you it's got a lot of research risk and you know it it might still very well fail know it it might still very well fail know it it might still very well fail but um I think that some amount of some but um I think that some amount of some but um I think that some amount of some very significant amount of research risk very significant amount of research risk very significant amount of research risk um was sort of put behind us when that um was sort of put behind us when that um was sort of put behind us when that started to work can you describe what started to work can you describe what started to work can you describe what kind of features can be extracted in kind of features can be extracted in kind of features can be extracted in this way well so it depends on the model this way well so it depends on the model this way well so it depends on the model that you're studying right so the the that you're studying right so the the that you're studying right so the the larger the model the more sophisticated larger the model the more sophisticated larger the model the more sophisticated they're going to be and we'll probably they're going to be and we'll probably they're going to be and we'll probably talk about about follow-up work in a talk about about follow-up work in a talk about about follow-up work in a minute but in these one layer models um minute but in these one layer models um minute but in these one layer models um so some very common things I think were so some very common things I think were so some very common things I think were were languages both programming were languages both programming were languages both programming languages and natural languages there languages and natural languages there languages and natural languages there were a lot of features that were um were a lot of features that were um were a lot of features that were um specific words in specific contexts so specific words in specific contexts so specific words in specific contexts so the and I think really the way to think the and I think really the way to think the and I think really the way to think about this is that the is likely about

about this is that the is likely about about this is that the is likely about to be followed by a noun so it's really to be followed by a noun so it's really to be followed by a noun so it's really you could think of this as the feature you could think of this as the feature you could think of this as the feature but you could also think of this as but you could also think of this as but you could also think of this as producting a specific noun feature and producting a specific noun feature and producting a specific noun feature and there would be these features that would there would be these features that would there would be these features that would fire for the in um the context of of say fire for the in um the context of of say fire for the in um the context of of say a legal document or a mathematical a legal document or a mathematical a legal document or a mathematical document or something something like document or something something like document or something something like this um and so uh you know maybe in the this um and so uh you know maybe in the this um and so uh you know maybe in the context of math you're like you know the context of math you're like you know the context of math you're like you know the and then predict Vector Matrix you know and then predict Vector Matrix you know and then predict Vector Matrix you know all these mathematical words whereas you all these mathematical words whereas you all these mathematical words whereas you other contexts you would predict other other contexts you would predict other other contexts you would predict other things that was that was common and things that was that was common and things that was that was common and basically we you need clever humans to basically we you need clever humans to basically we you need clever humans to assign labels to what we're seeing yes assign labels to what we're seeing yes assign labels to what we're seeing yes so you know this this is the only thing so you know this this is the only thing so you know this this is the only thing this is doing is that sort of um this is doing is that sort of um this is doing is that sort of um unfolding things for you so if unfolding things for you so if unfolding things for you so if everything was sort of folded over top everything was sort of folded over top everything was sort of folded over top of it you know cation folded everything of it you know cation folded everything of it you know cation folded everything on top of itself you can't really see it on top of itself you can't really see it on top of itself you can't really see it this is unfolding it but now you still this is unfolding it but now you still this is unfolding it but now you still have a very complex thing to try to have a very complex thing to try to have a very complex thing to try to understand um so then you have to do a understand um so then you have to do a understand um so then you have to do a bunch of work understanding what these bunch of work understanding what these bunch of work understanding what these are um and some of them are really are um and some of them are really are um and some of them are really subtle like there's some really cool subtle like there's some really cool subtle like there's some really cool things even in this this one layer model things even in this this one layer model things even in this this one layer model about um Unicode where you know of about um Unicode where you know of about um Unicode where you know of course some languages are in Unicode and course some languages are in Unicode and course some languages are in Unicode and the tokenizer won't necessarily have a the tokenizer won't necessarily have a the tokenizer won't necessarily have a dedicated token for every um Unicode um dedicated token for every um Unicode um dedicated token for every um Unicode um character so instead what you'll have is character so instead what you'll have is character so instead what you'll have is you'll have this these patterns of you'll have this these patterns of you'll have this these patterns of alternating token or alternating tokens alternating token or alternating tokens alternating token or alternating tokens that each represent half of a unic code that each represent half of a unic code that each represent half of a unic code character and then you have a different character and then you have a different character and then you have a different feature that you know goes and activates feature that you know goes and activates feature that you know goes and activates on the on the opposing ones to be like on the on the opposing ones to be like on the on the opposing ones to be like okay you know um I just finished a okay you know um I just finished a okay you know um I just finished a character you know go and predict the character you know go and predict the character you know go and predict the next prefix um then okay on the prefix next prefix um then okay on the prefix next prefix um then okay on the prefix you know predict a reasonable suffix um you know predict a reasonable suffix um you know predict a reasonable suffix um and you you have to alternate back and and you you have to alternate back and and you you have to alternate back and forth so there's you know these these

forth so there's you know these these forth so there's you know these these wer models are are really interesting wer models are are really interesting wer models are are really interesting and um uh I mean there's another thing and um uh I mean there's another thing and um uh I mean there's another thing which is you might think okay there which is you might think okay there which is you might think okay there would just be one b64 feature but it would just be one b64 feature but it would just be one b64 feature but it turns out there's actually a bunch of turns out there's actually a bunch of turns out there's actually a bunch of b64 features because you can have b64 features because you can have b64 features because you can have English text encoded in as b64 and that English text encoded in as b64 and that English text encoded in as b64 and that has a very different distribution of B has a very different distribution of B has a very different distribution of B 64 tokens than than regular and there's 64 tokens than than regular and there's 64 tokens than than regular and there's um uh there's there's some things about um uh there's there's some things about um uh there's there's some things about tokenization as well that it can exploit tokenization as well that it can exploit tokenization as well that it can exploit and I don't know there all all kinds of and I don't know there all all kinds of and I don't know there all all kinds of fun stuff how difficult is the task of fun stuff how difficult is the task of fun stuff how difficult is the task of sort of assigning labels to what's going sort of assigning labels to what's going sort of assigning labels to what's going on can this be automated by AI well I on can this be automated by AI well I on can this be automated by AI well I think it depends on the feature and it think it depends on the feature and it think it depends on the feature and it also depends on how much you trust your also depends on how much you trust your also depends on how much you trust your AI so um there's a lot of work doing um AI so um there's a lot of work doing um AI so um there's a lot of work doing um automated inability I think that's a automated inability I think that's a automated inability I think that's a really exciting Direction and we do a really exciting Direction and we do a really exciting Direction and we do a fair amount of automated inter and have fair amount of automated inter and have fair amount of automated inter and have have Claude go and label our features is have Claude go and label our features is have Claude go and label our features is there some fun moments where it's there some fun moments where it's there some fun moments where it's totally right or it's totally wrong yeah totally right or it's totally wrong yeah totally right or it's totally wrong yeah well I think I think it's very common well I think I think it's very common well I think I think it's very common that it's like says something very that it's like says something very that it's like says something very general which is like true in some sense general which is like true in some sense general which is like true in some sense but not really picking up on the but not really picking up on the but not really picking up on the specific of what's going on um so I specific of what's going on um so I specific of what's going on um so I think I think that's a pretty common think I think that's a pretty common think I think that's a pretty common situation um you don't know that I have situation um you don't know that I have situation um you don't know that I have a particularly amusing one that's a particularly amusing one that's a particularly amusing one that's interesting that little gap between it interesting that little gap between it interesting that little gap between it is true but it doesn't quite is true but it doesn't quite is true but it doesn't quite get to the Deep Nuance of a thing yeah get to the Deep Nuance of a thing yeah get to the Deep Nuance of a thing yeah that's a general challenge it's like that's a general challenge it's like that's a general challenge it's like it's it's St an incredible colish they it's it's St an incredible colish they it's it's St an incredible colish they can say a true thing but it doesn't it's can say a true thing but it doesn't it's can say a true thing but it doesn't it's qu it's not it's missing the depth qu it's not it's missing the depth qu it's not it's missing the depth sometimes and in this context it's like

sometimes and in this context it's like sometimes and in this context it's like the arc challenge you know the sort of the arc challenge you know the sort of the arc challenge you know the sort of IQ type tests it feels like figuring out IQ type tests it feels like figuring out IQ type tests it feels like figuring out what a feature represents is a bit of is what a feature represents is a bit of is what a feature represents is a bit of is a little puzzle you have to solve yeah a little puzzle you have to solve yeah a little puzzle you have to solve yeah and and I think that sometimes they're and and I think that sometimes they're and and I think that sometimes they're easier and sometimes they're harder as easier and sometimes they're harder as easier and sometimes they're harder as well um so well um so well um so uh yeah I think I think that's tricky uh yeah I think I think that's tricky uh yeah I think I think that's tricky now there's another thing which I don't now there's another thing which I don't now there's another thing which I don't know maybe maybe in some ways this is my know maybe maybe in some ways this is my know maybe maybe in some ways this is my like aesthetic coming in but I'll give like aesthetic coming in but I'll give like aesthetic coming in but I'll give try to give you a rationalization you try to give you a rationalization you try to give you a rationalization you know I'm actually a little suspicious of know I'm actually a little suspicious of know I'm actually a little suspicious of automated inability and I think that automated inability and I think that automated inability and I think that partly just that I want humans to partly just that I want humans to partly just that I want humans to understand neural net works and if the understand neural net works and if the understand neural net works and if the neural network is understanding it for neural network is understanding it for neural network is understanding it for me you know I'm I'm not I don't quite me you know I'm I'm not I don't quite me you know I'm I'm not I don't quite like that but I do have bit of a you like that but I do have bit of a you like that but I do have bit of a you know in some ways I'm sort of like the know in some ways I'm sort of like the know in some ways I'm sort of like the mathematicians who are like you know if mathematicians who are like you know if mathematicians who are like you know if there a computer automated proof it there a computer automated proof it there a computer automated proof it doesn't count U you know you they won't doesn't count U you know you they won't doesn't count U you know you they won't understand it but I I do also think that understand it but I I do also think that understand it but I I do also think that there is um this kind of like there is um this kind of like there is um this kind of like Reflections on trusting trust type issue Reflections on trusting trust type issue Reflections on trusting trust type issue where you know if you there's this where you know if you there's this where you know if you there's this famous talk about um uh you know you famous talk about um uh you know you famous talk about um uh you know you like when you're writing a computer like when you're writing a computer like when you're writing a computer program you have to trust your compiler program you have to trust your compiler program you have to trust your compiler and if there was like malware in your and if there was like malware in your and if there was like malware in your compiler then it could go and inject compiler then it could go and inject compiler then it could go and inject malware into the next compiler and you malware into the next compiler and you malware into the next compiler and you know you'd be kind of in trouble right know you'd be kind of in trouble right know you'd be kind of in trouble right well if you're using neural networks to well if you're using neural networks to well if you're using neural networks to go and um verify that your neural go and um verify that your neural go and um verify that your neural networks are safe the hypothesis that networks are safe the hypothesis that networks are safe the hypothesis that you're testing for is like okay well the you're testing for is like okay well the you're testing for is like okay well the neural network maybe isn't safe um and neural network maybe isn't safe um and neural network maybe isn't safe um and you have to worry about like is there you have to worry about like is there you have to worry about like is there some way that it could be screwing with some way that it could be screwing with some way that it could be screwing with you you you um so uh you know I I think that's not a um so uh you know I I think that's not a um so uh you know I I think that's not a big concern now um but I do Wonder in big concern now um but I do Wonder in big concern now um but I do Wonder in the long run if we have to use really the long run if we have to use really the long run if we have to use really powerful system AI systems to go and uh powerful system AI systems to go and uh powerful system AI systems to go and uh you know audit our AI systems is that is you know audit our AI systems is that is you know audit our AI systems is that is that actually something we can trust but that actually something we can trust but that actually something we can trust but maybe I'm just rationalizing because I I maybe I'm just rationalizing because I I maybe I'm just rationalizing because I I just want to us to have to get to a

just want to us to have to get to a just want to us to have to get to a point where humans understand everything point where humans understand everything point where humans understand everything yeah I mean especially that's hilarious yeah I mean especially that's hilarious yeah I mean especially that's hilarious especially as we talk about AI safety especially as we talk about AI safety especially as we talk about AI safety and it looking for features that would and it looking for features that would and it looking for features that would be relevant to AI safety like deception be relevant to AI safety like deception be relevant to AI safety like deception and so on uh so let's let's talk about and so on uh so let's let's talk about and so on uh so let's let's talk about the scaling a semanticity paper in May the scaling a semanticity paper in May the scaling a semanticity paper in May 2024 okay so what did it take to scale 2024 okay so what did it take to scale 2024 okay so what did it take to scale this to apply to Claude 3 on it well a this to apply to Claude 3 on it well a this to apply to Claude 3 on it well a lot of gpus a lot more gpus um but one lot of gpus a lot more gpus um but one lot of gpus a lot more gpus um but one of my teammates Tom henigan um was of my teammates Tom henigan um was of my teammates Tom henigan um was involved in the original scaling loss involved in the original scaling loss involved in the original scaling loss work um and something that he was sort work um and something that he was sort work um and something that he was sort of interested in from very early on is of interested in from very early on is of interested in from very early on is are there scaling laws for are there scaling laws for are there scaling laws for inability um and so um something he sort inability um and so um something he sort inability um and so um something he sort of immediately did when when this work of immediately did when when this work of immediately did when when this work started to succeed and we started to started to succeed and we started to started to succeed and we started to have sparse Auto encoders work we became have sparse Auto encoders work we became have sparse Auto encoders work we became very interested in you know what are the very interested in you know what are the very interested in you know what are the scaling laws for um uh you know for scaling laws for um uh you know for scaling laws for um uh you know for making making sparse Auto encoders making making sparse Auto encoders making making sparse Auto encoders larger and how does that relate to larger and how does that relate to larger and how does that relate to making the base model larger um and so making the base model larger um and so making the base model larger um and so um it turns out this works really well um it turns out this works really well um it turns out this works really well and you can use it to sort of project um and you can use it to sort of project um and you can use it to sort of project um you know if you train a sparse Auto you know if you train a sparse Auto you know if you train a sparse Auto encod a given size you know how many encod a given size you know how many encod a given size you know how many tokens should you train on and so on so tokens should you train on and so on so tokens should you train on and so on so this was actually a very big help to us this was actually a very big help to us this was actually a very big help to us in scaling up um this work um and made in scaling up um this work um and made in scaling up um this work um and made it a lot easier for us to go and train it a lot easier for us to go and train it a lot easier for us to go and train um you know really large sparse Auto um you know really large sparse Auto um you know really large sparse Auto encoders where you know um it's not like encoders where you know um it's not like encoders where you know um it's not like training the big models but it's it's training the big models but it's it's training the big models but it's it's starting to get to a point where it's starting to get to a point where it's starting to get to a point where it's actually actually expensive to go um and actually actually expensive to go um and actually actually expensive to go um and train the really big ones so you have to train the really big ones so you have to train the really big ones so you have to I mean you have to do all the stuff of I mean you have to do all the stuff of I mean you have to do all the stuff of like splitting it across large I mean like splitting it across large I mean like splitting it across large I mean there's a huge engineering challenge

there's a huge engineering challenge there's a huge engineering challenge here too right so yes so so there's here too right so yes so so there's here too right so yes so so there's there's a there's a scientific question there's a there's a scientific question there's a there's a scientific question of how do you scale things effectively of how do you scale things effectively of how do you scale things effectively um and then there's an enormous amount um and then there's an enormous amount um and then there's an enormous amount of engineering to go and scale this up of engineering to go and scale this up of engineering to go and scale this up you have to you have to chart it you you have to you have to chart it you you have to you have to chart it you have to you have to think very carefully have to you have to think very carefully have to you have to think very carefully about a lot of things I'm lucky to work about a lot of things I'm lucky to work about a lot of things I'm lucky to work with a bunch of great Engineers cuz I am with a bunch of great Engineers cuz I am with a bunch of great Engineers cuz I am definitely not a great engine yeah on definitely not a great engine yeah on definitely not a great engine yeah on the infrastructure especially yeah for the infrastructure especially yeah for the infrastructure especially yeah for sure so it turns out tldr it worked it sure so it turns out tldr it worked it sure so it turns out tldr it worked it worked yeah and and I think this is worked yeah and and I think this is worked yeah and and I think this is important because you could have important because you could have important because you could have imagined you could like you could have imagined you could like you could have imagined you could like you could have imagined a world where you set after imagined a world where you set after imagined a world where you set after towards monos fanticy you know Chris towards monos fanticy you know Chris towards monos fanticy you know Chris this is great you know it works on a one this is great you know it works on a one this is great you know it works on a one layer model but one layer models are layer model but one layer models are layer model but one layer models are really idiosyncratic um like you know really idiosyncratic um like you know really idiosyncratic um like you know maybe maybe there just something ID like maybe maybe there just something ID like maybe maybe there just something ID like maybe the linear representation maybe the linear representation maybe the linear representation hypothesis and super hypothesis is the hypothesis and super hypothesis is the hypothesis and super hypothesis is the right way to understand a one layer right way to understand a one layer right way to understand a one layer model but it's not the right way to model but it's not the right way to model but it's not the right way to understand large models um and so I understand large models um and so I understand large models um and so I think um I mean first of all like The think um I mean first of all like The think um I mean first of all like The Cutting him at all paper sort of um cut Cutting him at all paper sort of um cut Cutting him at all paper sort of um cut through that a little bit and and sort through that a little bit and and sort through that a little bit and and sort of suggested that this wasn't the case of suggested that this wasn't the case of suggested that this wasn't the case but um scaling onity sort of I think was but um scaling onity sort of I think was but um scaling onity sort of I think was significant evidence that even for very significant evidence that even for very significant evidence that even for very large models and we did it on Claude 3 large models and we did it on Claude 3 large models and we did it on Claude 3 sauna which at that point was uh one of sauna which at that point was uh one of sauna which at that point was uh one of our production models um you know even our production models um you know even our production models um you know even these models um seem to be very you know these models um seem to be very you know these models um seem to be very you know seem to be substantially explained at seem to be substantially explained at seem to be substantially explained at least by linear features and you know least by linear features and you know least by linear features and you know doing dictionary learning on them works doing dictionary learning on them works doing dictionary learning on them works and as you learn more features you go and as you learn more features you go and as you learn more features you go and you explain explain more and more so and you explain explain more and more so and you explain explain more and more so that's a I think a quite a promising that's a I think a quite a promising that's a I think a quite a promising sign and you find now really fascinating sign and you find now really fascinating sign and you find now really fascinating abstract features um and the features abstract features um and the features abstract features um and the features are also multimodal they respond to are also multimodal they respond to are also multimodal they respond to images and text for the same concept images and text for the same concept images and text for the same concept which is fun yeah this can you explain which is fun yeah this can you explain which is fun yeah this can you explain that I mean like you know back door

that I mean like you know back door that I mean like you know back door there's just a lot of examples that you there's just a lot of examples that you there's just a lot of examples that you can yeah so maybe maybe let's start with can yeah so maybe maybe let's start with can yeah so maybe maybe let's start with a one example to start which is we found a one example to start which is we found a one example to start which is we found some features around sort of security some features around sort of security some features around sort of security vulnerabilities and back doors and codes vulnerabilities and back doors and codes vulnerabilities and back doors and codes so it turns out those are actually two so it turns out those are actually two so it turns out those are actually two different features um so there's a different features um so there's a different features um so there's a security vulnerability feature and if security vulnerability feature and if security vulnerability feature and if you force it active Claude will start to you force it active Claude will start to you force it active Claude will start to go and write um security vulnerabilities go and write um security vulnerabilities go and write um security vulnerabilities like buffer overflows into code and it like buffer overflows into code and it like buffer overflows into code and it also it fires for all kinds of things also it fires for all kinds of things also it fires for all kinds of things like you know some of some of the top like you know some of some of the top like you know some of some of the top data set examples for it were things data set examples for it were things data set examples for it were things like you know dash dash disable um you like you know dash dash disable um you like you know dash dash disable um you know SSL or something like this which know SSL or something like this which know SSL or something like this which are sort of obviously really um uh are sort of obviously really um uh are sort of obviously really um uh really insecure so at this point it's really insecure so at this point it's really insecure so at this point it's kind of like maybe it's just because the kind of like maybe it's just because the kind of like maybe it's just because the examples are presented that way it's examples are presented that way it's examples are presented that way it's kind of like surface a little bit more kind of like surface a little bit more kind of like surface a little bit more obvious examples right um I guess the obvious examples right um I guess the obvious examples right um I guess the the idea is that down the line might be the idea is that down the line might be the idea is that down the line might be able to detect more Nuance like able to detect more Nuance like able to detect more Nuance like deception or bugs or that kind of stuff deception or bugs or that kind of stuff deception or bugs or that kind of stuff yeah well I maybe I want to distinguish yeah well I maybe I want to distinguish yeah well I maybe I want to distinguish two things so um one is um the two things so um one is um the two things so um one is um the complexity of the feature or the concept complexity of the feature or the concept complexity of the feature or the concept right and the other is right and the other is right and the other is the the Nuance of the how subtle the the the Nuance of the how subtle the the the Nuance of the how subtle the examples we're looking at right so when examples we're looking at right so when examples we're looking at right so when we when we show the top data set we when we show the top data set we when we show the top data set examples those are the most extreme examples those are the most extreme examples those are the most extreme examples that that feature to to examples that that feature to to examples that that feature to to activate um and so it doesn't mean that activate um and so it doesn't mean that activate um and so it doesn't mean that it doesn't fire for more subtle things it doesn't fire for more subtle things it doesn't fire for more subtle things so the UN you know the insecure um code so the UN you know the insecure um code so the UN you know the insecure um code feature you know the stuff that it fires feature you know the stuff that it fires feature you know the stuff that it fires for most strongly for are these like for most strongly for are these like for most strongly for are these like really obvious you know disable the really obvious you know disable the really obvious you know disable the security type things um but um um you security type things um but um um you security type things um but um um you know uh it it also Fires for you know

know uh it it also Fires for you know know uh it it also Fires for you know buffer overflows and and more subtle buffer overflows and and more subtle buffer overflows and and more subtle security vulnerabilities in code you security vulnerabilities in code you security vulnerabilities in code you know these features are all multimodal know these features are all multimodal know these features are all multimodal so you could ask like what images so you could ask like what images so you could ask like what images activate this feature and it turns out activate this feature and it turns out activate this feature and it turns out um that the uh the the security um that the uh the the security um that the uh the the security vulnerability feature activates for vulnerability feature activates for vulnerability feature activates for images of um uh like people clicking on images of um uh like people clicking on images of um uh like people clicking on Chrome to like go past the like you know Chrome to like go past the like you know Chrome to like go past the like you know this this website uh the SSL certificate this this website uh the SSL certificate this this website uh the SSL certificate might be wrong or something like this might be wrong or something like this might be wrong or something like this another thing that's very entertaining another thing that's very entertaining another thing that's very entertaining is there's backd doors en code feature is there's backd doors en code feature is there's backd doors en code feature like you activate it it goes and Cloud like you activate it it goes and Cloud like you activate it it goes and Cloud writes a back door that like will go and writes a back door that like will go and writes a back door that like will go and dump your data to port or something but dump your data to port or something but dump your data to port or something but you can ask okay what what images you can ask okay what what images you can ask okay what what images activate the back door feature it was activate the back door feature it was activate the back door feature it was devices with hidden cameras in them so devices with hidden cameras in them so devices with hidden cameras in them so there's a whole apparently genre of there's a whole apparently genre of there's a whole apparently genre of people going and selling devices that people going and selling devices that people going and selling devices that look in uous that have hidden cameras look in uous that have hidden cameras look in uous that have hidden cameras and they have ads that how there's a and they have ads that how there's a and they have ads that how there's a hidden camera in it and I guess that is hidden camera in it and I guess that is hidden camera in it and I guess that is the you know physical version of a back the you know physical version of a back the you know physical version of a back door um and so it sort of shows you how door um and so it sort of shows you how door um and so it sort of shows you how abstract these concepts are right um and abstract these concepts are right um and abstract these concepts are right um and I I just thought that was uh I I'm sort I I just thought that was uh I I'm sort I I just thought that was uh I I'm sort of sad that there's a whole Market of of sad that there's a whole Market of of sad that there's a whole Market of people selling devices like that but I people selling devices like that but I people selling devices like that but I was kind of delighted that that was the was kind of delighted that that was the was kind of delighted that that was the the thing that it came up with as the the thing that it came up with as the the thing that it came up with as the the top uh image examples for the the top uh image examples for the the top uh image examples for the feature yeah it's nice it's multimodal feature yeah it's nice it's multimodal feature yeah it's nice it's multimodal it's multi almost context it's it's as it's multi almost context it's it's as it's multi almost context it's it's as broad strong definition of a singular broad strong definition of a singular broad strong definition of a singular concept it's nice yeah to me one of the concept it's nice yeah to me one of the concept it's nice yeah to me one of the really interesting features especially really interesting features especially really interesting features especially for AI safety is deception and lying and for AI safety is deception and lying and for AI safety is deception and lying and the possibility that these kinds of the possibility that these kinds of the possibility that these kinds of methods could detect uh lying in a model methods could detect uh lying in a model methods could detect uh lying in a model especially gets smarter and smarter and especially gets smarter and smarter and especially gets smarter and smarter and smarter presumably that's a big threat smarter presumably that's a big threat smarter presumably that's a big threat of a super intelligent model that he can

of a super intelligent model that he can of a super intelligent model that he can deceive the people operating deceive the people operating deceive the people operating it uh as to its intentions or any of it uh as to its intentions or any of it uh as to its intentions or any of that kind of stuff so what what have you that kind of stuff so what what have you that kind of stuff so what what have you learned from detecting lying inside learned from detecting lying inside learned from detecting lying inside models yeah so I think we're in some models yeah so I think we're in some models yeah so I think we're in some ways in early days for that we find ways in early days for that we find ways in early days for that we find quite a few features related to quite a few features related to quite a few features related to deception and lying there's one feature deception and lying there's one feature deception and lying there's one feature where fires for people lying and being where fires for people lying and being where fires for people lying and being deceptive and you force it active and deceptive and you force it active and deceptive and you force it active and Claude starts lying to you so we have a Claude starts lying to you so we have a Claude starts lying to you so we have a have a deception feature I mean there's have a deception feature I mean there's have a deception feature I mean there's all kinds of other features about all kinds of other features about all kinds of other features about withholding information and not withholding information and not withholding information and not answering questions features about power answering questions features about power answering questions features about power seeking and coups and stuff like that seeking and coups and stuff like that seeking and coups and stuff like that this a lot of features that are kind of this a lot of features that are kind of this a lot of features that are kind of related to Spooky things and if you um related to Spooky things and if you um related to Spooky things and if you um force them active Claude will will force them active Claude will will force them active Claude will will behave in ways that are they're not the behave in ways that are they're not the behave in ways that are they're not the kind of behaviors you want what are kind of behaviors you want what are kind of behaviors you want what are possible next exciting directions to you possible next exciting directions to you possible next exciting directions to you in the space of uh Mech and well there's in the space of uh Mech and well there's in the space of uh Mech and well there's a lot of things um so for one thing I would really like um so for one thing I would really like to get to a point where we have circuits to get to a point where we have circuits to get to a point where we have circuits where we can really understand um not where we can really understand um not where we can really understand um not just the features uh but then use that just the features uh but then use that just the features uh but then use that to understand the computation of models to understand the computation of models to understand the computation of models um that really for me is is the the um that really for me is is the the um that really for me is is the the ultimate goal of this um and there's ultimate goal of this um and there's ultimate goal of this um and there's been some work we we put out a few been some work we we put out a few been some work we we put out a few things there's a paper from Sam Marks things there's a paper from Sam Marks things there's a paper from Sam Marks that does some stuff like this there's that does some stuff like this there's that does some stuff like this there's been some I'd say some work around the been some I'd say some work around the been some I'd say some work around the edges here um but I think there's a lot edges here um but I think there's a lot edges here um but I think there's a lot more to do and I think that will be a more to do and I think that will be a more to do and I think that will be a very exciting thing um that's related to very exciting thing um that's related to very exciting thing um that's related to a challenge we call interference weights

a challenge we call interference weights a challenge we call interference weights um where um due to supersition if you um where um due to supersition if you um where um due to supersition if you just sort of navely look at whether just sort of navely look at whether just sort of navely look at whether featur are connected together there may featur are connected together there may featur are connected together there may be some weights that sort of don't exist be some weights that sort of don't exist be some weights that sort of don't exist in the upstairs model but are just sort in the upstairs model but are just sort in the upstairs model but are just sort of artifacts of of superposition so of artifacts of of superposition so of artifacts of of superposition so that's a a sort of technical challenge that's a a sort of technical challenge that's a a sort of technical challenge related to that related to that related to that um I think another exciting direction is um I think another exciting direction is um I think another exciting direction is just I you know you might think of of just I you know you might think of of just I you know you might think of of sparse Auto encoders as being kind of sparse Auto encoders as being kind of sparse Auto encoders as being kind of like a telescope they allow us to you like a telescope they allow us to you like a telescope they allow us to you know look out and see all these features know look out and see all these features know look out and see all these features that are are are are out there and you that are are are are out there and you that are are are are out there and you know as we build better and better know as we build better and better know as we build better and better sparse Auto en Cutters get better better sparse Auto en Cutters get better better sparse Auto en Cutters get better better at dictionary learning we see more and at dictionary learning we see more and at dictionary learning we see more and more stars um and you know we zoom in on more stars um and you know we zoom in on more stars um and you know we zoom in on smaller and smaller stars but there kind smaller and smaller stars but there kind smaller and smaller stars but there kind of um a lot of evidence that we're only of um a lot of evidence that we're only of um a lot of evidence that we're only still seeing a very small fraction of still seeing a very small fraction of still seeing a very small fraction of the Stars there's a lot of matter in our the Stars there's a lot of matter in our the Stars there's a lot of matter in our in our you know neural network universe in our you know neural network universe in our you know neural network universe that we can't observe yet um and it may that we can't observe yet um and it may that we can't observe yet um and it may be that um that we'll never be able to be that um that we'll never be able to be that um that we'll never be able to have fine enough instruments to observe have fine enough instruments to observe have fine enough instruments to observe it and maybe maybe some of it just isn't it and maybe maybe some of it just isn't it and maybe maybe some of it just isn't possible um isn't computationally possible um isn't computationally possible um isn't computationally tractable to observant there's sort of a tractable to observant there's sort of a tractable to observant there's sort of a a kind of dark matter and in not in a kind of dark matter and in not in a kind of dark matter and in not in maybe the sense of of astronomy of maybe the sense of of astronomy of maybe the sense of of astronomy of earlier astronomy when we didn't know earlier astronomy when we didn't know earlier astronomy when we didn't know what this unexplained matter is um and what this unexplained matter is um and what this unexplained matter is um and so I I think a lot about that that dark so I I think a lot about that that dark so I I think a lot about that that dark matter and whether will ever observe it matter and whether will ever observe it matter and whether will ever observe it and what that means for safety if we if and what that means for safety if we if and what that means for safety if we if we can't observe it if there's you know we can't observe it if there's you know we can't observe it if there's you know some if some significant fraction of nor some if some significant fraction of nor some if some significant fraction of nor networks are not accessible to us um networks are not accessible to us um networks are not accessible to us um another question that I think a lot another question that I think a lot another question that I think a lot about is uh at the end of the day you about is uh at the end of the day you about is uh at the end of the day you know mechanistic inter is it's very know mechanistic inter is it's very know mechanistic inter is it's very microscopic um approach to interality

microscopic um approach to interality microscopic um approach to interality it's trying to understand things in a it's trying to understand things in a it's trying to understand things in a very fine grained way but lot of the very fine grained way but lot of the very fine grained way but lot of the questions we care about are very questions we care about are very questions we care about are very macroscopic um you know we we care about macroscopic um you know we we care about macroscopic um you know we we care about these questions about neural network these questions about neural network these questions about neural network behavior and behavior and behavior and and I think that's the thing that I care and I think that's the thing that I care and I think that's the thing that I care most about but there's there's lots of most about but there's there's lots of most about but there's there's lots of other other sort of larger scale other other sort of larger scale other other sort of larger scale questions you you might care about um questions you you might care about um questions you you might care about um and somehow you know the nice thing and somehow you know the nice thing and somehow you know the nice thing about about having a very microscopic about about having a very microscopic about about having a very microscopic approach is it's maybe easier to ask you approach is it's maybe easier to ask you approach is it's maybe easier to ask you know is this true but the downside is know is this true but the downside is know is this true but the downside is it's much further from the things we it's much further from the things we it's much further from the things we care about and so we now have this care about and so we now have this care about and so we now have this ladder to climb and I think there's a ladder to climb and I think there's a ladder to climb and I think there's a question of can will we be able to find question of can will we be able to find question of can will we be able to find are there are there sort of larger scale are there are there sort of larger scale are there are there sort of larger scale abstractions that we can use to abstractions that we can use to abstractions that we can use to understand nural networks that can we understand nural networks that can we understand nural networks that can we get up from this very microscopic get up from this very microscopic get up from this very microscopic approach yeah you've you you've written approach yeah you've you you've written approach yeah you've you you've written about this this kind of organs question about this this kind of organs question about this this kind of organs question yeah exactly if we uh think of yeah exactly if we uh think of yeah exactly if we uh think of interpretability as a kind of anatomy of interpretability as a kind of anatomy of interpretability as a kind of anatomy of neural networks most of the circus neural networks most of the circus neural networks most of the circus threads involve studying tiny little threads involve studying tiny little threads involve studying tiny little veins looking at the small scale and veins looking at the small scale and veins looking at the small scale and individual neurons and how they connect individual neurons and how they connect individual neurons and how they connect however there are many natural questions however there are many natural questions however there are many natural questions that the small scale approach doesn't that the small scale approach doesn't that the small scale approach doesn't address in contrast the most prominent address in contrast the most prominent address in contrast the most prominent abstractions in biological Anatomy abstractions in biological Anatomy abstractions in biological Anatomy involve larger scale structures like involve larger scale structures like involve larger scale structures like individual organs like the heart or individual organs like the heart or individual organs like the heart or entire organ systems like the entire organ systems like the entire organ systems like the respiratory system and so we wonder is respiratory system and so we wonder is respiratory system and so we wonder is there a respiratory system or heart or there a respiratory system or heart or there a respiratory system or heart or brain region of an artificial neuron brain region of an artificial neuron brain region of an artificial neuron Network yeah exactly um and I mean like Network yeah exactly um and I mean like Network yeah exactly um and I mean like if you think about science right a lot if you think about science right a lot if you think about science right a lot of scientific Fields have um you know of scientific Fields have um you know of scientific Fields have um you know investigate things that many level of

investigate things that many level of investigate things that many level of abstractions in biology you have like abstractions in biology you have like abstractions in biology you have like you know molecular biology studying you you know molecular biology studying you you know molecular biology studying you know proteins and molecules and so on know proteins and molecules and so on know proteins and molecules and so on and you have cellular biology and then and you have cellular biology and then and you have cellular biology and then you have histology studying tissues and you have histology studying tissues and you have histology studying tissues and you have anatomy and then you have you have anatomy and then you have you have anatomy and then you have zoology and then you have ecology and so zoology and then you have ecology and so zoology and then you have ecology and so you have many many levels of abstraction you have many many levels of abstraction you have many many levels of abstraction or you know physics maybe the physics of or you know physics maybe the physics of or you know physics maybe the physics of individual particles and then you know individual particles and then you know individual particles and then you know statistical physics gives you gives you statistical physics gives you gives you statistical physics gives you gives you thermodynamics and things like this and thermodynamics and things like this and thermodynamics and things like this and so you often have different levels of so you often have different levels of so you often have different levels of abstraction um and I think that right abstraction um and I think that right abstraction um and I think that right now we have you know mechanistic now we have you know mechanistic now we have you know mechanistic interpret if it succeeds is sort of like interpret if it succeeds is sort of like interpret if it succeeds is sort of like a microbiology of neural networks but we a microbiology of neural networks but we a microbiology of neural networks but we we want something more like anatomy and we want something more like anatomy and we want something more like anatomy and so and you know a question you might ask so and you know a question you might ask so and you know a question you might ask is why why can't you just go there is why why can't you just go there is why why can't you just go there directly and I think the answer is super directly and I think the answer is super directly and I think the answer is super um in at least in significant part it's um in at least in significant part it's um in at least in significant part it's that it's actually very hard to to see that it's actually very hard to to see that it's actually very hard to to see this this macroscopic structure U this this macroscopic structure U this this macroscopic structure U without first sort of breaking down the without first sort of breaking down the without first sort of breaking down the microscopic structure in the right way microscopic structure in the right way microscopic structure in the right way and then studying how it connects and then studying how it connects and then studying how it connects together um but I'm I'm hopeful that together um but I'm I'm hopeful that together um but I'm I'm hopeful that there is going to be something much there is going to be something much there is going to be something much larger than um features and circuits and larger than um features and circuits and larger than um features and circuits and that we're going to be able to have a that we're going to be able to have a that we're going to be able to have a story that's much than evolves much story that's much than evolves much story that's much than evolves much bigger things and you then you can sort bigger things and you then you can sort bigger things and you then you can sort of study in detail the parts you care of study in detail the parts you care of study in detail the parts you care about as opposed to neurobiology like a about as opposed to neurobiology like a about as opposed to neurobiology like a psychologist or psychiatrist when your psychologist or psychiatrist when your psychologist or psychiatrist when your own network and I think that the own network and I think that the own network and I think that the beautiful thing would be if we could go beautiful thing would be if we could go beautiful thing would be if we could go and rather than having disperate fields and rather than having disperate fields and rather than having disperate fields for those two things if you could have a for those two things if you could have a for those two things if you could have a build a bridge between them such that build a bridge between them such that build a bridge between them such that you could go and um uh have all of your you could go and um uh have all of your you could go and um uh have all of your higher level abstractions be grounded higher level abstractions be grounded higher level abstractions be grounded very firmly In This Very solid um you

very firmly In This Very solid um you very firmly In This Very solid um you know more rigorous ideally Foundation know more rigorous ideally Foundation know more rigorous ideally Foundation what do you think is the difference what do you think is the difference what do you think is the difference between the human brain the biological between the human brain the biological between the human brain the biological neuron Network and the artificial neuron neuron Network and the artificial neuron neuron Network and the artificial neuron Network well the neuroscientists have a Network well the neuroscientists have a Network well the neuroscientists have a much harder job than us you know much harder job than us you know much harder job than us you know sometimes I just like count my blessings sometimes I just like count my blessings sometimes I just like count my blessings by how much easier my job is than the by how much easier my job is than the by how much easier my job is than the neuroscientist right so I have um we we neuroscientist right so I have um we we neuroscientist right so I have um we we can record from all the neurons yeah we can record from all the neurons yeah we can record from all the neurons yeah we can do that on arbitrary amounts of data can do that on arbitrary amounts of data can do that on arbitrary amounts of data um the neurons don't change while you're um the neurons don't change while you're um the neurons don't change while you're doing that by the way MH um you can go doing that by the way MH um you can go doing that by the way MH um you can go and ablate neurons you can edit the and ablate neurons you can edit the and ablate neurons you can edit the connections and so on um and then you connections and so on um and then you connections and so on um and then you undo those changes that's prettyy great undo those changes that's prettyy great undo those changes that's prettyy great yeah um you can force any you can yeah um you can force any you can yeah um you can force any you can intervene on any neuron and force it intervene on any neuron and force it intervene on any neuron and force it active and see what happens um you know active and see what happens um you know active and see what happens um you know which neurons are connected to which neurons are connected to which neurons are connected to everything right you neuroscientists everything right you neuroscientists everything right you neuroscientists want to get the connecto we have the want to get the connecto we have the want to get the connecto we have the connecto um and we have it for like much connecto um and we have it for like much connecto um and we have it for like much bigger than the elegant um and then not bigger than the elegant um and then not bigger than the elegant um and then not only do we have the connectome um we only do we have the connectome um we only do we have the connectome um we know uh what the you know which neurons know uh what the you know which neurons know uh what the you know which neurons excite or inhibit each other right so we excite or inhibit each other right so we excite or inhibit each other right so we have we it's not just that we know that have we it's not just that we know that have we it's not just that we know that like the binary mask we know the the like the binary mask we know the the like the binary mask we know the the weights um we can take gradients we know weights um we can take gradients we know weights um we can take gradients we know computationally what each neuron does um computationally what each neuron does um computationally what each neuron does um so I don't know the goes on and on we so I don't know the goes on and on we so I don't know the goes on and on we just have um so many advantages over just have um so many advantages over just have um so many advantages over neuroscientists and then despite having neuroscientists and then despite having neuroscientists and then despite having all those advantages it's really hard all those advantages it's really hard all those advantages it's really hard and so one thing I do sometimes think is and so one thing I do sometimes think is and so one thing I do sometimes think is like gosh like if it's this hard for us like gosh like if it's this hard for us like gosh like if it's this hard for us it seems impossible under the it seems impossible under the it seems impossible under the constraints of Neuroscience or you know constraints of Neuroscience or you know constraints of Neuroscience or you know near impossible um I I I don't know near impossible um I I I don't know near impossible um I I I don't know maybe maybe part of me is like I've got maybe maybe part of me is like I've got maybe maybe part of me is like I've got a few neuroscientists on my team maybe a few neuroscientists on my team maybe a few neuroscientists on my team maybe maybe I'm sort of like ah you know um maybe I'm sort of like ah you know um maybe I'm sort of like ah you know um the uh maybe the neuroscientists maybe the uh maybe the neuroscientists maybe the uh maybe the neuroscientists maybe some of them would like to have an some of them would like to have an some of them would like to have an easier problem that's still very hard um

easier problem that's still very hard um easier problem that's still very hard um and they they could come and work on on and they they could come and work on on and they they could come and work on on neural networks and then after we after neural networks and then after we after neural networks and then after we after we figure out things in sort of the easy we figure out things in sort of the easy we figure out things in sort of the easy uh Little Pond of trying to understand uh Little Pond of trying to understand uh Little Pond of trying to understand neural networks which is still very hard neural networks which is still very hard neural networks which is still very hard then we then we could go back to then we then we could go back to then we then we could go back to biological Neuroscience I love what biological Neuroscience I love what biological Neuroscience I love what you've written about the goal of mechan you've written about the goal of mechan you've written about the goal of mechan turp research as uh two goals safety and turp research as uh two goals safety and turp research as uh two goals safety and Beauty so can you talk about the beauty Beauty so can you talk about the beauty Beauty so can you talk about the beauty side of things yeah so you know there's side of things yeah so you know there's side of things yeah so you know there's this funny thing where I think some this funny thing where I think some this funny thing where I think some people want uh some people are kind of people want uh some people are kind of people want uh some people are kind of disappointed by neural networks I think disappointed by neural networks I think disappointed by neural networks I think where they're like ah you know neural where they're like ah you know neural where they're like ah you know neural network network network um it's these just these simple rules um it's these just these simple rules um it's these just these simple rules then you just like do a bunch of then you just like do a bunch of then you just like do a bunch of engineering to scale it up and it works engineering to scale it up and it works engineering to scale it up and it works really well and like where's the like really well and like where's the like really well and like where's the like complex ideas you know this isn't like a complex ideas you know this isn't like a complex ideas you know this isn't like a very nice beautiful scientific very nice beautiful scientific very nice beautiful scientific result and I sometimes think when people result and I sometimes think when people result and I sometimes think when people say that I picture them being like you say that I picture them being like you say that I picture them being like you know evolution is so boring it's just a know evolution is so boring it's just a know evolution is so boring it's just a bunch of simple rules and you run bunch of simple rules and you run bunch of simple rules and you run Evolution for a long time and you get Evolution for a long time and you get Evolution for a long time and you get biology like what a what a a sucky uh biology like what a what a a sucky uh biology like what a what a a sucky uh you know way for biology to have turned you know way for biology to have turned you know way for biology to have turned out where's the the complex rules but out where's the the complex rules but out where's the the complex rules but the beauty is that the Simplicity the beauty is that the Simplicity the beauty is that the Simplicity generates complexity um you know biology generates complexity um you know biology generates complexity um you know biology has these simple rules and it gives rise has these simple rules and it gives rise has these simple rules and it gives rise to you know all the life and ecosystems to you know all the life and ecosystems to you know all the life and ecosystems that we see around us all the beauty of that we see around us all the beauty of that we see around us all the beauty of nature that all just comes from nature that all just comes from nature that all just comes from Evolution and from something very simple Evolution and from something very simple Evolution and from something very simple Evolution and similarly I think that Evolution and similarly I think that Evolution and similarly I think that nural networks build you know create nural networks build you know create nural networks build you know create enormous um complexity and Beauty inside enormous um complexity and Beauty inside enormous um complexity and Beauty inside and structure inside themselves that and structure inside themselves that and structure inside themselves that people generally don't look at and don't people generally don't look at and don't people generally don't look at and don't try to understand because it's it's hard try to understand because it's it's hard try to understand because it's it's hard to understand but I I think that there to understand but I I think that there to understand but I I think that there is an Inc incredibly Rich structure to

is an Inc incredibly Rich structure to is an Inc incredibly Rich structure to be discovered inside n networks a lot of be discovered inside n networks a lot of be discovered inside n networks a lot of a lot of very deep Beauty um if we're a lot of very deep Beauty um if we're a lot of very deep Beauty um if we're just willing to take the time to go and just willing to take the time to go and just willing to take the time to go and see it and understand it yeah I love I see it and understand it yeah I love I see it and understand it yeah I love I love Mech inter the feeling like we are love Mech inter the feeling like we are love Mech inter the feeling like we are understanding or getting glimpses of understanding or getting glimpses of understanding or getting glimpses of understanding the magic that's going on understanding the magic that's going on understanding the magic that's going on inside is really wonderful it feels to inside is really wonderful it feels to inside is really wonderful it feels to me like one of the questions is just me like one of the questions is just me like one of the questions is just calling out to be asked and I'm sort of calling out to be asked and I'm sort of calling out to be asked and I'm sort of I mean a lot of people are think about I mean a lot of people are think about I mean a lot of people are think about this but I'm often surprised that morar this but I'm often surprised that morar this but I'm often surprised that morar is how is it that we don't know how to is how is it that we don't know how to is how is it that we don't know how to create computer systems that can do create computer systems that can do create computer systems that can do these things and yet we have these these things and yet we have these these things and yet we have these amazing systems that we don't know how amazing systems that we don't know how amazing systems that we don't know how to directly create computer programs to directly create computer programs to directly create computer programs that can do these things but these that can do these things but these that can do these things but these neural networks can do all these amazing neural networks can do all these amazing neural networks can do all these amazing things and it just feels like that is things and it just feels like that is things and it just feels like that is obviously the question that sort of is obviously the question that sort of is obviously the question that sort of is calling out to be answered if you are if calling out to be answered if you are if calling out to be answered if you are if you have any degree of curiosity it's you have any degree of curiosity it's you have any degree of curiosity it's it's like how is it that that Humanity it's like how is it that that Humanity it's like how is it that that Humanity now has these artifacts that can do now has these artifacts that can do now has these artifacts that can do these things that we don't know how to these things that we don't know how to these things that we don't know how to do yeah I love the image of the circus do yeah I love the image of the circus do yeah I love the image of the circus towards the light of the objective towards the light of the objective towards the light of the objective function yeah it's just it's it's this function yeah it's just it's it's this function yeah it's just it's it's this organic thing that we've grown and we organic thing that we've grown and we organic thing that we've grown and we have no idea what we've grown well thank have no idea what we've grown well thank have no idea what we've grown well thank you for working on safety and thank you you for working on safety and thank you you for working on safety and thank you for appreciating the beauty of the for appreciating the beauty of the for appreciating the beauty of the things you uh discover and thank you for things you uh discover and thank you for things you uh discover and thank you for talking today Chris this is wonderful talking today Chris this is wonderful talking today Chris this is wonderful thank you for taking the time to chat as thank you for taking the time to chat as thank you for taking the time to chat as well thanks for listening to this well thanks for listening to this well thanks for listening to this conversation with Chris Ola and before conversation with Chris Ola and before conversation with Chris Ola and before that with DAR amade and Amanda ascal to that with DAR amade and Amanda ascal to that with DAR amade and Amanda ascal to support this podcast please check out support this podcast please check out support this podcast please check out our sponsors in the description and now our sponsors in the description and now our sponsors in the description and now let me leave you with some words from let me leave you with some words from let me leave you with some words from Alan Watts Alan Watts Alan Watts the only way to make sense out of change

the only way to make sense out of change the only way to make sense out of change is to plunge into it move with it and is to plunge into it move with it and is to plunge into it move with it and join the join the join the dance thank you for listening and hope dance thank you for listening and hope dance thank you for listening and hope to see you next time