Max Tegmark: The Case for Halting AI Development | Lex Fridman Podcast #371

a lot of people have said for many years a lot of people have said for many years that there will come a time when they that there will come a time when they that there will come a time when they want to pause a little bit want to pause a little bit want to pause a little bit that time is now the following is a conversation with Max the following is a conversation with Max tegmark his third time in the podcast in tegmark his third time in the podcast in tegmark his third time in the podcast in fact his first appearance was episode fact his first appearance was episode fact his first appearance was episode number one of this very podcast he is a number one of this very podcast he is a number one of this very podcast he is a physicist and artificial intelligence physicist and artificial intelligence physicist and artificial intelligence researcher at MIT co-founder of future researcher at MIT co-founder of future researcher at MIT co-founder of future Life Institute and author of Life 3.0 Life Institute and author of Life 3.0 Life Institute and author of Life 3.0 Being Human in the age of artificial Being Human in the age of artificial Being Human in the age of artificial intelligence intelligence intelligence most recently he's a key figure in most recently he's a key figure in most recently he's a key figure in spearheading the open letter calling for spearheading the open letter calling for spearheading the open letter calling for a six-month pause on giant AI a six-month pause on giant AI a six-month pause on giant AI experiments like training gpt4 the experiments like training gpt4 the experiments like training gpt4 the letter reads letter reads letter reads were calling for a pause on training of were calling for a pause on training of were calling for a pause on training of models larger than GPT 4 for 6 months models larger than GPT 4 for 6 months models larger than GPT 4 for 6 months this does not imply a pause or ban on this does not imply a pause or ban on this does not imply a pause or ban on all AI research and development or the all AI research and development or the all AI research and development or the use of systems that have already been use of systems that have already been use of systems that have already been placed on the market our call is placed on the market our call is placed on the market our call is specific and addresses a very small pool specific and addresses a very small pool specific and addresses a very small pool of actors who possesses this capability of actors who possesses this capability of actors who possesses this capability the letter has been signed by over 50 the letter has been signed by over 50 the letter has been signed by over 50 000 individuals including 1800 CEOs and 000 individuals including 1800 CEOs and 000 individuals including 1800 CEOs and over 1500 professors signatories include over 1500 professors signatories include over 1500 professors signatories include Joshua bengio Stewart Russell Elon Musk Joshua bengio Stewart Russell Elon Musk Joshua bengio Stewart Russell Elon Musk Steve Wozniak you all know a Harari Steve Wozniak you all know a Harari Steve Wozniak you all know a Harari Andrew Yang and many others Andrew Yang and many others Andrew Yang and many others this is a defining moment in the history this is a defining moment in the history this is a defining moment in the history of human civilization or the balance of of human civilization or the balance of of human civilization or the balance of power between human and AI begins to power between human and AI begins to power between human and AI begins to shift

shift shift and Max's mind and his voice is one of and Max's mind and his voice is one of and Max's mind and his voice is one of the most valuable and Powerful in a time the most valuable and Powerful in a time the most valuable and Powerful in a time like this like this like this his support his wisdom his friendship his support his wisdom his friendship his support his wisdom his friendship has been a gift of forever deeply has been a gift of forever deeply has been a gift of forever deeply grateful for grateful for grateful for this is the Lex Friedman podcast to this is the Lex Friedman podcast to this is the Lex Friedman podcast to support it please check out our sponsors support it please check out our sponsors support it please check out our sponsors in the description and now dear friends in the description and now dear friends in the description and now dear friends here's Max Ted mark here's Max Ted mark here's Max Ted mark you were the first ever guest on this you were the first ever guest on this you were the first ever guest on this podcast episode number one so first of podcast episode number one so first of podcast episode number one so first of all all all Max I just have to say uh thank you for Max I just have to say uh thank you for Max I just have to say uh thank you for giving me a chance thank you for giving me a chance thank you for giving me a chance thank you for starting this journey it's been an starting this journey it's been an starting this journey it's been an incredible journey just thank you for um incredible journey just thank you for um incredible journey just thank you for um sitting down with me and just acting sitting down with me and just acting sitting down with me and just acting like I'm somebody who matters that I'm like I'm somebody who matters that I'm like I'm somebody who matters that I'm somebody who's interesting to talk to somebody who's interesting to talk to somebody who's interesting to talk to and uh thank you for doing it I meant a and uh thank you for doing it I meant a and uh thank you for doing it I meant a lot all right thanks to you for putting lot all right thanks to you for putting lot all right thanks to you for putting your heart and soul into this I know your heart and soul into this I know your heart and soul into this I know when you delve into controversial topics when you delve into controversial topics when you delve into controversial topics it's inevitable to get hit by what what it's inevitable to get hit by what what it's inevitable to get hit by what what Hamlet talks about the slings and arrows Hamlet talks about the slings and arrows Hamlet talks about the slings and arrows and stuff and I really admire this it's and stuff and I really admire this it's and stuff and I really admire this it's in an era you know where YouTube videos in an era you know where YouTube videos in an era you know where YouTube videos are too long and now it has to be like a are too long and now it has to be like a are too long and now it has to be like a 20 minute Tick Tock 20 second Tick Tock 20 minute Tick Tock 20 second Tick Tock 20 minute Tick Tock 20 second Tick Tock clip it's just so refreshing to see you clip it's just so refreshing to see you clip it's just so refreshing to see you going exactly against all of the advice going exactly against all of the advice going exactly against all of the advice and doing this with this really long and doing this with this really long and doing this with this really long form things and that people appreciate form things and that people appreciate form things and that people appreciate it you know reality is nuanced and uh it you know reality is nuanced and uh it you know reality is nuanced and uh thanks for thanks for thanks for and sharing it that way uh so let me ask and sharing it that way uh so let me ask and sharing it that way uh so let me ask you again the first question I've ever you again the first question I've ever you again the first question I've ever asked on this podcast episode number one asked on this podcast episode number one asked on this podcast episode number one talking to you do you think there's

talking to you do you think there's talking to you do you think there's intelligent life out there in the intelligent life out there in the intelligent life out there in the universe let's revisit that question do universe let's revisit that question do universe let's revisit that question do you have any updates you have any updates you have any updates what's your view what's your view what's your view when you look out to the Stars so when when you look out to the Stars so when when you look out to the Stars so when we look after the Stars we look after the Stars we look after the Stars if you define our universe the way most if you define our universe the way most if you define our universe the way most astrophysicists do not this all of space astrophysicists do not this all of space astrophysicists do not this all of space but the spherical region of space that but the spherical region of space that but the spherical region of space that we can see with our telescopes from we can see with our telescopes from we can see with our telescopes from which light has the time to reach us which light has the time to reach us which light has the time to reach us since our big bang since our big bang since our big bang I'm in the minority I I'm I'm in the minority I I'm I'm in the minority I I'm estimate that we are the only life estimate that we are the only life estimate that we are the only life in this spherical volume that has uh in this spherical volume that has uh in this spherical volume that has uh invented Internet radios gotten our invented Internet radios gotten our invented Internet radios gotten our level of tech and um if that's true level of tech and um if that's true level of tech and um if that's true then it puts a lot of responsibility on then it puts a lot of responsibility on then it puts a lot of responsibility on us to not mess this one up because if us to not mess this one up because if us to not mess this one up because if it's true it means that life is is quite it's true it means that life is is quite it's true it means that life is is quite rare and we are stewards of this one rare and we are stewards of this one rare and we are stewards of this one spark of advanced Consciousness which if spark of advanced Consciousness which if spark of advanced Consciousness which if we nurture it then we nurture it then we nurture it then help it grow it immensely life can help it grow it immensely life can help it grow it immensely life can spread from here out into much of our spread from here out into much of our spread from here out into much of our universe and we can have this just universe and we can have this just universe and we can have this just amazing future whereas if we instead um amazing future whereas if we instead um amazing future whereas if we instead um are Reckless with the technology we are Reckless with the technology we are Reckless with the technology we build and just snuff it out due to the build and just snuff it out due to the build and just snuff it out due to the stupidity stupidity stupidity or in fighting then or in fighting then or in fighting then maybe the rest of cosmic history in our maybe the rest of cosmic history in our maybe the rest of cosmic history in our universe was just going to be a play for universe was just going to be a play for universe was just going to be a play for empty benches but I I do think empty benches but I I do think empty benches but I I do think that we are actually very likely to get that we are actually very likely to get that we are actually very likely to get visited by aliens visited by aliens visited by aliens alien intelligence quite soon but I

alien intelligence quite soon but I alien intelligence quite soon but I think we are going to be building that think we are going to be building that think we are going to be building that alien intelligence alien intelligence alien intelligence so so so uh we're going to give birth uh we're going to give birth uh we're going to give birth to an intelligent alien civilization to an intelligent alien civilization to an intelligent alien civilization unlike anything that human the evolution unlike anything that human the evolution unlike anything that human the evolution here on Earth was able to create in here on Earth was able to create in here on Earth was able to create in terms of the path the biological path it terms of the path the biological path it terms of the path the biological path it took yeah and it's gonna be much more took yeah and it's gonna be much more took yeah and it's gonna be much more alien than alien than alien than cats or even the most exotic animal on cats or even the most exotic animal on cats or even the most exotic animal on the planet right now because it will not the planet right now because it will not the planet right now because it will not have been created through the usual have been created through the usual have been created through the usual darwinian competition where it darwinian competition where it darwinian competition where it necessarily cares about necessarily cares about necessarily cares about self-preservation is afraid of death self-preservation is afraid of death self-preservation is afraid of death um any of those things um any of those things um any of those things the space of alien Minds is just that the space of alien Minds is just that the space of alien Minds is just that you can build it's just so much faster you can build it's just so much faster you can build it's just so much faster than what evolution will give you than what evolution will give you than what evolution will give you and with that also comes great and with that also comes great and with that also comes great responsibility but the the for us to responsibility but the the for us to responsibility but the the for us to make sure that the kind of Minds we make sure that the kind of Minds we make sure that the kind of Minds we create are those kind of Minds that um create are those kind of Minds that um create are those kind of Minds that um it's good to create Minds that will uh it's good to create Minds that will uh it's good to create Minds that will uh share our values and and be good for share our values and and be good for share our values and and be good for Humanity and life and also mine don't Humanity and life and also mine don't Humanity and life and also mine don't create Minds that don't suffer create Minds that don't suffer create Minds that don't suffer do you try to visualize the full space do you try to visualize the full space do you try to visualize the full space of alien Minds that AI could be to try of alien Minds that AI could be to try of alien Minds that AI could be to try to consider all the different kinds of to consider all the different kinds of to consider all the different kinds of intelligences intelligences intelligences sort of generalizing what humans are sort of generalizing what humans are sort of generalizing what humans are able to do to the full spectrum of an able to do to the full spectrum of an able to do to the full spectrum of an intelligent creatures entities could do intelligent creatures entities could do intelligent creatures entities could do I try but I would say I fail I mean it's

I try but I would say I fail I mean it's I try but I would say I fail I mean it's it's very difficult for a human mind really grapple with really grapple with something still completely alien even something still completely alien even something still completely alien even even for us right if we just try to even for us right if we just try to even for us right if we just try to imagine how would it feel if we were imagine how would it feel if we were imagine how would it feel if we were completely indifferent towards death or completely indifferent towards death or completely indifferent towards death or individuality if we even if you just individuality if we even if you just individuality if we even if you just imagine that for example you could just copy my knowledge of how you could just copy my knowledge of how to speak Swedish boom now you can speak to speak Swedish boom now you can speak to speak Swedish boom now you can speak Swedish Swedish Swedish and you could copy any of my cool and you could copy any of my cool and you could copy any of my cool experiences and then you could delete experiences and then you could delete experiences and then you could delete the ones you didn't like in your own the ones you didn't like in your own the ones you didn't like in your own life just like that it life just like that it life just like that it it would already change quite a lot it would already change quite a lot it would already change quite a lot about how you feel as a human being about how you feel as a human being about how you feel as a human being right if you probably spend less effort right if you probably spend less effort right if you probably spend less effort studying things if you just copy them studying things if you just copy them studying things if you just copy them and you might be less afraid of death and you might be less afraid of death and you might be less afraid of death because if the plane you're on starts to because if the plane you're on starts to because if the plane you're on starts to crash you'd just be like oh shucks I'm crash you'd just be like oh shucks I'm crash you'd just be like oh shucks I'm gonna I haven't backed my brain up for gonna I haven't backed my brain up for gonna I haven't backed my brain up for four hours so I'm gonna lose this all four hours so I'm gonna lose this all four hours so I'm gonna lose this all this wonderful experiences of this this wonderful experiences of this this wonderful experiences of this flight flight flight we might also start feeling we might also start feeling we might also start feeling more like compassionate maybe with other more like compassionate maybe with other more like compassionate maybe with other people if we can so readily share each people if we can so readily share each people if we can so readily share each other's experiences in our knowledge and other's experiences in our knowledge and other's experiences in our knowledge and feel more like a hive mind it's very feel more like a hive mind it's very feel more like a hive mind it's very hard though I I really hard though I I really hard though I I really feel very humble about this feel very humble about this feel very humble about this to to Grapple with it that the how it to to Grapple with it that the how it to to Grapple with it that the how it might actually feel that the the one might actually feel that the the one might actually feel that the the one thing which is so obvious though it's I thing which is so obvious though it's I thing which is so obvious though it's I think is just really worth reflecting on

think is just really worth reflecting on think is just really worth reflecting on is because the mind space of possible is because the mind space of possible is because the mind space of possible intelligence is so different from ours intelligence is so different from ours intelligence is so different from ours it's very dangerous if we assume they're it's very dangerous if we assume they're it's very dangerous if we assume they're going to be like us or anything like us going to be like us or anything like us going to be like us or anything like us well there's a well there's a well there's a the entirety of uh human written history the entirety of uh human written history the entirety of uh human written history has been through poetry through novels has been through poetry through novels has been through poetry through novels been trying to describe her philosophy been trying to describe her philosophy been trying to describe her philosophy uh try and describe the Human Condition uh try and describe the Human Condition uh try and describe the Human Condition and what's entailed in it like just like and what's entailed in it like just like and what's entailed in it like just like you said fear of death and all those you said fear of death and all those you said fear of death and all those kinds of things what is love and all of kinds of things what is love and all of kinds of things what is love and all of that changes yeah if you have a that changes yeah if you have a that changes yeah if you have a different kind of intelligence yeah like different kind of intelligence yeah like different kind of intelligence yeah like all of it the entirety all those poems all of it the entirety all those poems all of it the entirety all those poems they're trying to sneak up to what the they're trying to sneak up to what the they're trying to sneak up to what the hell it means to be human all of that hell it means to be human all of that hell it means to be human all of that changes how AI concerns and uh changes how AI concerns and uh changes how AI concerns and uh existential crises that AI experiences existential crises that AI experiences existential crises that AI experiences how that clashes with the human how that clashes with the human how that clashes with the human existential crisis The Human Condition existential crisis The Human Condition existential crisis The Human Condition yeah that's hard to hard to Fathom how yeah that's hard to hard to Fathom how yeah that's hard to hard to Fathom how to predict it's hard but it's to predict it's hard but it's to predict it's hard but it's fascinating to think about also even in fascinating to think about also even in fascinating to think about also even in the best case scenario where we don't the best case scenario where we don't the best case scenario where we don't lose control of over the ever more lose control of over the ever more lose control of over the ever more powerful AI that we're building to powerful AI that we're building to powerful AI that we're building to other humans whose goals we think are other humans whose goals we think are other humans whose goals we think are horrible and where we don't lose control horrible and where we don't lose control horrible and where we don't lose control to the machines to the machines to the machines and AI and AI and AI provides the things we want even then provides the things we want even then provides the things we want even then you get into the questions do you you get into the questions do you you get into the questions do you touched here you know maybe it's the touched here you know maybe it's the touched here you know maybe it's the struggle that it's actually hard to do struggle that it's actually hard to do struggle that it's actually hard to do things is part of the things that gives things is part of the things that gives things is part of the things that gives us meaning as well right so for example us meaning as well right so for example us meaning as well right so for example I found it so shocking that this new

I found it so shocking that this new I found it so shocking that this new Microsoft gpt4 commercial that they put Microsoft gpt4 commercial that they put Microsoft gpt4 commercial that they put together has this woman talking about together has this woman talking about together has this woman talking about and showing this demo how she's going to and showing this demo how she's going to and showing this demo how she's going to give a graduation speech to her beloved give a graduation speech to her beloved give a graduation speech to her beloved daughter and she asks gpt4 to write it daughter and she asks gpt4 to write it daughter and she asks gpt4 to write it it was freaking 200 words or so if I it was freaking 200 words or so if I it was freaking 200 words or so if I realized that my parents couldn't be realized that my parents couldn't be realized that my parents couldn't be bothered struggling a little bit to bothered struggling a little bit to bothered struggling a little bit to write write write 200 words and Outsource that to their 200 words and Outsource that to their 200 words and Outsource that to their computer I would feel really offended computer I would feel really offended computer I would feel really offended actually actually actually and so I wonder if um eliminating too and so I wonder if um eliminating too and so I wonder if um eliminating too much of the struggle from our existence much of the struggle from our existence much of the struggle from our existence what what what do you think that would also take away a do you think that would also take away a do you think that would also take away a little bit of little bit of little bit of what it means to be human yeah what it means to be human yeah what it means to be human yeah we can't even predict I had somebody we can't even predict I had somebody we can't even predict I had somebody mentioned to me that they use they mentioned to me that they use they mentioned to me that they use they started using uh started using uh started using uh GPT with a 3.5 not 4.0 GPT with a 3.5 not 4.0 GPT with a 3.5 not 4.0 uh to write what they really feel to a uh to write what they really feel to a uh to write what they really feel to a person person person and they have a temper issue and they're and they have a temper issue and they're and they have a temper issue and they're basically trying to get Chad gbt to basically trying to get Chad gbt to basically trying to get Chad gbt to rewrite it in a nicer way to get the rewrite it in a nicer way to get the rewrite it in a nicer way to get the point across but we write in a nicer way point across but we write in a nicer way point across but we write in a nicer way so we're even removing the inner so we're even removing the inner so we're even removing the inner from our communication so I don't you from our communication so I don't you from our communication so I don't you know there's some positive know there's some positive know there's some positive aspects of that but mostly it's just the aspects of that but mostly it's just the aspects of that but mostly it's just the transformation of how humans communicate transformation of how humans communicate transformation of how humans communicate and it's scary because so much of our and it's scary because so much of our and it's scary because so much of our society is based on this glue of

society is based on this glue of society is based on this glue of communication and if that we're now communication and if that we're now communication and if that we're now using AI as the medium of communication using AI as the medium of communication using AI as the medium of communication that that does the language for us uh so that that does the language for us uh so that that does the language for us uh so much of the emotion that's Laden in much of the emotion that's Laden in much of the emotion that's Laden in human communication so much of the human communication so much of the human communication so much of the intent intent intent that's going to be handled by an that's going to be handled by an that's going to be handled by an outsourced AI how does that change outsourced AI how does that change outsourced AI how does that change everything how how does it change the everything how how does it change the everything how how does it change the internal state of how we feel about internal state of how we feel about internal state of how we feel about other human beings what makes us lonely other human beings what makes us lonely other human beings what makes us lonely what makes us excited yeah what makes us what makes us excited yeah what makes us what makes us excited yeah what makes us afraid how we fall in love all that kind afraid how we fall in love all that kind afraid how we fall in love all that kind of stuff yeah for me personally I have of stuff yeah for me personally I have of stuff yeah for me personally I have to confess the challenge is one of the to confess the challenge is one of the to confess the challenge is one of the things that really makes my life feel things that really makes my life feel things that really makes my life feel meaningful you know meaningful you know meaningful you know if I go hiking mountain with my wife if I go hiking mountain with my wife if I go hiking mountain with my wife Maya I don't want to just press a button Maya I don't want to just press a button Maya I don't want to just press a button and be at the top but I want to struggle and be at the top but I want to struggle and be at the top but I want to struggle and come up there sweaty and feel wow we and come up there sweaty and feel wow we and come up there sweaty and feel wow we did this in the same way did this in the same way did this in the same way I want to constantly work on myself to I want to constantly work on myself to I want to constantly work on myself to become a better person if I say become a better person if I say become a better person if I say something in Anger that I regret I want something in Anger that I regret I want something in Anger that I regret I want to go back and and really work on myself to go back and and really work on myself to go back and and really work on myself rather than just tell an AI just from rather than just tell an AI just from rather than just tell an AI just from now on always filter what I write so I now on always filter what I write so I now on always filter what I write so I don't have to work on myself because don't have to work on myself because don't have to work on myself because then I'm not growing then I'm not growing then I'm not growing yeah but then again it could be like yeah but then again it could be like yeah but then again it could be like with chess and AI with chess and AI with chess and AI wants it significantly obviously wants it significantly obviously wants it significantly obviously supersedes the performance of humans it supersedes the performance of humans it supersedes the performance of humans it will live in its own world and provide will live in its own world and provide will live in its own world and provide maybe a uh flourishing Civilizations for maybe a uh flourishing Civilizations for maybe a uh flourishing Civilizations for humans but we humans will continue

humans but we humans will continue humans but we humans will continue hiking mountains and playing our games hiking mountains and playing our games hiking mountains and playing our games even though AI is so much smarter so even though AI is so much smarter so even though AI is so much smarter so much stronger so much Superior in every much stronger so much Superior in every much stronger so much Superior in every single way just like with chess yeah so single way just like with chess yeah so single way just like with chess yeah so that that I mean that's one possible that that I mean that's one possible that that I mean that's one possible hopeful trajectory here is that humans hopeful trajectory here is that humans hopeful trajectory here is that humans will continue to Human will continue to Human will continue to Human uh and AI will just be a uh and AI will just be a uh and AI will just be a kind of a a medium that enables The Human a a medium that enables The Human Experience to flourish yeah Experience to flourish yeah Experience to flourish yeah I would phrase that as rebranding I would phrase that as rebranding I would phrase that as rebranding ourselves from Homo sapiens to homo ourselves from Homo sapiens to homo ourselves from Homo sapiens to homo sentience you know right now with sentience you know right now with sentience you know right now with sapiens the ability to be intelligent sapiens the ability to be intelligent sapiens the ability to be intelligent we've even put it in our species name we've even put it in our species name we've even put it in our species name we're branding ourselves as the smartest we're branding ourselves as the smartest we're branding ourselves as the smartest yeah information processing yeah information processing yeah information processing entity on the planet that's clearly entity on the planet that's clearly entity on the planet that's clearly gonna gonna gonna change if AI continues ahead change if AI continues ahead change if AI continues ahead so maybe we should focus on the so maybe we should focus on the so maybe we should focus on the experience instead the subjective experience instead the subjective experience instead the subjective experience that we have with experience that we have with experience that we have with homo sentience and and so that's what's homo sentience and and so that's what's homo sentience and and so that's what's really valuable the love the connection really valuable the love the connection really valuable the love the connection the other things and get off our high horses and get rid and get off our high horses and get rid of this hubris that only we can of this hubris that only we can of this hubris that only we can do we do integrals so Consciousness and do we do integrals so Consciousness and do we do integrals so Consciousness and subjective experience is a fundamental subjective experience is a fundamental subjective experience is a fundamental value value value to what it means to be human make that to what it means to be human make that to what it means to be human make that make that the priority

make that the priority make that the priority that feels like a hopeful direction to that feels like a hopeful direction to that feels like a hopeful direction to me but that also requires more me but that also requires more me but that also requires more compassion not just towards other humans compassion not just towards other humans compassion not just towards other humans because they happen to be the smartest because they happen to be the smartest because they happen to be the smartest on the planet but also towards all our on the planet but also towards all our on the planet but also towards all our other fellow creatures on this planet other fellow creatures on this planet other fellow creatures on this planet and I I personally feel right now we're and I I personally feel right now we're and I I personally feel right now we're treating a lot of farm animals horribly treating a lot of farm animals horribly treating a lot of farm animals horribly for example and the excuse we're using for example and the excuse we're using for example and the excuse we're using is oh they're not as smart as us is oh they're not as smart as us is oh they're not as smart as us but if we get that we're not that smart but if we get that we're not that smart but if we get that we're not that smart in the grand scheme of things either in in the grand scheme of things either in in the grand scheme of things either in the post aie Epoch you know the post aie Epoch you know the post aie Epoch you know then surely we should value then surely we should value then surely we should value the subjective experience of a cow also the subjective experience of a cow also the subjective experience of a cow also well well well allow me to briefly look at the book allow me to briefly look at the book allow me to briefly look at the book which at this point is becoming more and which at this point is becoming more and which at this point is becoming more and more Visionary that you've written I more Visionary that you've written I more Visionary that you've written I guess over five years ago life 3.0 guess over five years ago life 3.0 guess over five years ago life 3.0 so first of all 3.0 what's 1.0 what's so first of all 3.0 what's 1.0 what's so first of all 3.0 what's 1.0 what's 2.0 was 3.0 and how's that Vision sort 2.0 was 3.0 and how's that Vision sort 2.0 was 3.0 and how's that Vision sort of evolve the vision in the book evolved of evolve the vision in the book evolved of evolve the vision in the book evolved to today life 1.0 is really dumb like to today life 1.0 is really dumb like to today life 1.0 is really dumb like bacteria and that it can't actually bacteria and that it can't actually bacteria and that it can't actually learn anything at all during the learn anything at all during the learn anything at all during the lifetime the learning just comes from lifetime the learning just comes from lifetime the learning just comes from this genetic this genetic this genetic process from one generation to the next process from one generation to the next process from one generation to the next Life 2.0 Is Us and other animals which Life 2.0 Is Us and other animals which Life 2.0 Is Us and other animals which have brains which can learn during their have brains which can learn during their have brains which can learn during their lifetime a great deal right so and um you know you were born without and um you know you were born without being able to speak English and at some being able to speak English and at some being able to speak English and at some point you decided hey I want to upgrade point you decided hey I want to upgrade point you decided hey I want to upgrade my software let's install an

my software let's install an my software let's install an English-speaking module English-speaking module English-speaking module so you did so you did so you did and life 3.0 does not exist yet can and life 3.0 does not exist yet can and life 3.0 does not exist yet can cannot replace not only its software the cannot replace not only its software the cannot replace not only its software the way we can but also it's Hardware way we can but also it's Hardware way we can but also it's Hardware and um that's where we're heading and um that's where we're heading and um that's where we're heading towards at high speed we're already towards at high speed we're already towards at high speed we're already maybe 2.1 because we can you know put in maybe 2.1 because we can you know put in maybe 2.1 because we can you know put in a a a an artificial knee uh pacemaker an artificial knee uh pacemaker an artificial knee uh pacemaker etc etc and if newer Link in other etc etc and if newer Link in other etc etc and if newer Link in other companies succeed will be like 2.2 Etc companies succeed will be like 2.2 Etc companies succeed will be like 2.2 Etc but uh well the Company's trying to but uh well the Company's trying to but uh well the Company's trying to build AGI are trying to make is of build AGI are trying to make is of build AGI are trying to make is of course full 3.0 and you can put that course full 3.0 and you can put that course full 3.0 and you can put that intelligence into something that also intelligence into something that also intelligence into something that also has no biological basis whatsoever so let's biological basis whatsoever so let's constraints and more capabilities just constraints and more capabilities just constraints and more capabilities just like the leap from 1.0 to 2.0 there is like the leap from 1.0 to 2.0 there is like the leap from 1.0 to 2.0 there is nevertheless he's speaking so harshly nevertheless he's speaking so harshly nevertheless he's speaking so harshly about bacteria so disrespectfully about about bacteria so disrespectfully about about bacteria so disrespectfully about bacteria there is still the same kind of bacteria there is still the same kind of bacteria there is still the same kind of magic there magic there magic there that permeates Life 2.0 and uh and 3.0 that permeates Life 2.0 and uh and 3.0 that permeates Life 2.0 and uh and 3.0 it seems like maybe the thing that's it seems like maybe the thing that's it seems like maybe the thing that's truly powerful truly powerful truly powerful about life intelligence and about life intelligence and about life intelligence and Consciousness was already there in 1.0 Consciousness was already there in 1.0 Consciousness was already there in 1.0 is it possible is it possible is it possible I think we should be humble and not be I think we should be humble and not be I think we should be humble and not be so quick so quick so quick make everything binary and say either make everything binary and say either make everything binary and say either it's there or it's not clearly there's a it's there or it's not clearly there's a it's there or it's not clearly there's a there's a great spectrum and there is there's a great spectrum and there is there's a great spectrum and there is even controversy by whether some

even controversy by whether some even controversy by whether some unicellular or organisms like amoebas unicellular or organisms like amoebas unicellular or organisms like amoebas can maybe learn a little bit can maybe learn a little bit can maybe learn a little bit you know after all so apologies if I you know after all so apologies if I you know after all so apologies if I offended anything there yeah it wasn't offended anything there yeah it wasn't offended anything there yeah it wasn't by intent it was more that I wanted to by intent it was more that I wanted to by intent it was more that I wanted to talk up how cool it is to actually have talk up how cool it is to actually have talk up how cool it is to actually have a brain yeah where you can learn a brain yeah where you can learn a brain yeah where you can learn dramatically within your lifetime dramatically within your lifetime dramatically within your lifetime typical human and and the higher up you typical human and and the higher up you typical human and and the higher up you get from 1.0 to 2.0 to 3.0 the more you get from 1.0 to 2.0 to 3.0 the more you get from 1.0 to 2.0 to 3.0 the more you become the captain of your own desk of become the captain of your own desk of become the captain of your own desk of your own ship the master of your own your own ship the master of your own your own ship the master of your own destiny and the less you become a slave destiny and the less you become a slave destiny and the less you become a slave to whatever Evolution gave you right to whatever Evolution gave you right to whatever Evolution gave you right by upgrading our software which can be by upgrading our software which can be by upgrading our software which can be so different from previous generations so different from previous generations so different from previous generations and even from our parents and even from our parents and even from our parents much more so than even a bacterium you much more so than even a bacterium you much more so than even a bacterium you know no offense to them know no offense to them know no offense to them and if you can also swap out your and if you can also swap out your and if you can also swap out your Hardware take any physical form you want Hardware take any physical form you want Hardware take any physical form you want of course it's really the sky's the of course it's really the sky's the of course it's really the sky's the limit limit limit yeah so the yeah so the yeah so the it accelerates the rate at which you can it accelerates the rate at which you can it accelerates the rate at which you can perform the competition computation that perform the competition computation that perform the competition computation that determines your destiny determines your destiny determines your destiny yeah and I think it's it's worth yeah and I think it's it's worth yeah and I think it's it's worth commenting a bit on what you means in commenting a bit on what you means in commenting a bit on what you means in this context also if you swap things out this context also if you swap things out this context also if you swap things out a lot right now a lot right now a lot right now this is controversial but my current current understanding is that that you know understanding is that that you know understanding is that that you know life is best thought of not as a bag of life is best thought of not as a bag of life is best thought of not as a bag of meat or even a bag of meat or even a bag of meat or even a bag of Elementary particles but rather as in as Elementary particles but rather as in as Elementary particles but rather as in as um um um a system which can process information a system which can process information a system which can process information and retain its own complexity

and retain its own complexity and retain its own complexity even though nature is always trying to even though nature is always trying to even though nature is always trying to mess it up so mess it up so mess it up so it's all about information processing it's all about information processing it's all about information processing and and and that makes it a lot like something like that makes it a lot like something like that makes it a lot like something like a wave in the ocean which is not it's a wave in the ocean which is not it's a wave in the ocean which is not it's it's water molecules right the water it's water molecules right the water it's water molecules right the water molecules bob up and down but the wave molecules bob up and down but the wave molecules bob up and down but the wave moves forward it's an information moves forward it's an information moves forward it's an information pattern in the same way you Lex pattern in the same way you Lex pattern in the same way you Lex you're not the same atoms as during the you're not the same atoms as during the you're not the same atoms as during the first time you did with me you've first time you did with me you've first time you did with me you've swapped out most of them but still you swapped out most of them but still you swapped out most of them but still you yeah and yeah and yeah and the the information pattern is still the the information pattern is still the the information pattern is still there and um there and um there and um if you if you could swap out your arms if you if you could swap out your arms if you if you could swap out your arms and whatever you can still have this kind of you can still have this kind of continuity it becomes much more continuity it becomes much more continuity it becomes much more sophisticated sort of way before in time sophisticated sort of way before in time sophisticated sort of way before in time where the information lives on I I lost where the information lives on I I lost where the information lives on I I lost both of my parents since since our last both of my parents since since our last both of my parents since since our last podcast and and it actually gives me a podcast and and it actually gives me a podcast and and it actually gives me a lot of Solace that lot of Solace that lot of Solace that this way of thinking about them this way of thinking about them this way of thinking about them they haven't entirely died because they haven't entirely died because they haven't entirely died because a lot of mommy and daddy's um a lot of mommy and daddy's um a lot of mommy and daddy's um sorry I'm getting a little emotional sorry I'm getting a little emotional sorry I'm getting a little emotional here but a lot of their values here but a lot of their values here but a lot of their values and ideas and even jokes and so on they and ideas and even jokes and so on they and ideas and even jokes and so on they haven't gone away right some of them haven't gone away right some of them haven't gone away right some of them live on I can carry on some of them and live on I can carry on some of them and live on I can carry on some of them and they also live on a lot of other and a they also live on a lot of other and a they also live on a lot of other and a lot of other people so in this sense lot of other people so in this sense lot of other people so in this sense even with Life 2.0 we can to some extent even with Life 2.0 we can to some extent even with Life 2.0 we can to some extent already transcend already transcend already transcend our physical bodies and our death our physical bodies and our death our physical bodies and our death and particularly if you can share your

and particularly if you can share your and particularly if you can share your own information your own ideas with many own information your own ideas with many own information your own ideas with many others like you do in your podcast others like you do in your podcast others like you do in your podcast then um then um then um you know that's the closest immortality you know that's the closest immortality you know that's the closest immortality we can get with our biobodies you carry we can get with our biobodies you carry we can get with our biobodies you carry a little bit of them in you yes yeah a little bit of them in you yes yeah a little bit of them in you yes yeah uh do you miss them you miss your mom uh do you miss them you miss your mom uh do you miss them you miss your mom and dad of course of course what did you and dad of course of course what did you and dad of course of course what did you learn about life from them if it can learn about life from them if it can learn about life from them if it can take a bit take a bit take a bit of a tangent of a tangent of a tangent on so many things on so many things on so many things um for starters my my Fascination for Math for starters my my Fascination for Math and and and um the physical mysteries of our um the physical mysteries of our um the physical mysteries of our University thinking I got a lot of that University thinking I got a lot of that University thinking I got a lot of that for my dad but I think my obsession for for my dad but I think my obsession for for my dad but I think my obsession for really big questions and Consciousness really big questions and Consciousness really big questions and Consciousness and so on that actually came mostly for and so on that actually came mostly for and so on that actually came mostly for my mom my mom my mom and and and when I got from both of them which is when I got from both of them which is when I got from both of them which is very core part of really who I am I very core part of really who I am I very core part of really who I am I think is think is think is is is is um um um this um not buying into what everybody else is not buying into what everybody else is saying just saying just saying just dude what I think is right they both they both very much just you know did their own very much just you know did their own very much just you know did their own thing and sometimes they got flagged for thing and sometimes they got flagged for thing and sometimes they got flagged for it and it did it anyway

it and it did it anyway it and it did it anyway that's why you've always been an that's why you've always been an that's why you've always been an inspiration to me that you're at the top inspiration to me that you're at the top inspiration to me that you're at the top of your field and you still of your field and you still of your field and you still you still willing to uh you still willing to uh you still willing to uh to tackle the big questions in your own to tackle the big questions in your own to tackle the big questions in your own way you're one of the one of the people way you're one of the one of the people way you're one of the one of the people that represents that represents that represents MIT best to me you've always been an MIT best to me you've always been an MIT best to me you've always been an inspiration in that so it's good to hear inspiration in that so it's good to hear inspiration in that so it's good to hear that you got that from your mom and dad that you got that from your mom and dad that you got that from your mom and dad yeah you're too kind but but yeah I mean yeah you're too kind but but yeah I mean yeah you're too kind but but yeah I mean the real the good reason to do science the real the good reason to do science the real the good reason to do science is because you're really curious you is because you're really curious you is because you're really curious you want to figure out the truth want to figure out the truth want to figure out the truth if you think if you think if you think this is how it is and everyone else says this is how it is and everyone else says this is how it is and everyone else says no no that's and it's that way no no that's and it's that way no no that's and it's that way you know you know you know you sticked with what you think is true you sticked with what you think is true you sticked with what you think is true and and and and and and even if even if even if everybody else keeps thinking it's everybody else keeps thinking it's everybody else keeps thinking it's there's a certain there's a certain there's a certain um um um I always root for the underdog when I I always root for the underdog when I I always root for the underdog when I watch movies and my my dad once I I one watch movies and my my dad once I I one watch movies and my my dad once I I one time for example when I wrote one of my time for example when I wrote one of my time for example when I wrote one of my craziest papers ever or I'm talking craziest papers ever or I'm talking craziest papers ever or I'm talking about our universe ultimately being about our universe ultimately being about our universe ultimately being mathematical which we're not going to mathematical which we're not going to mathematical which we're not going to get into today I got this email from a get into today I got this email from a get into today I got this email from a quite famous Professor saying this is quite famous Professor saying this is quite famous Professor saying this is not only but it's going to ruin not only but it's going to ruin not only but it's going to ruin your career you should stop doing this your career you should stop doing this your career you should stop doing this kind of stuff I sent it to my dad do you kind of stuff I sent it to my dad do you kind of stuff I sent it to my dad do you know what he said what'd he say he know what he said what'd he say he know what he said what'd he say he replied with a quote from Dante segil Tu replied with a quote from Dante segil Tu replied with a quote from Dante segil Tu Corso Corso Corso follow your own path and let the people follow your own path and let the people follow your own path and let the people talk talk talk go Dad yeah this is the kind of thing go Dad yeah this is the kind of thing go Dad yeah this is the kind of thing you know he's dead but that that you know he's dead but that that you know he's dead but that that attitude is not

attitude is not attitude is not how did losing them as a man as a human how did losing them as a man as a human how did losing them as a man as a human being change you being change you being change you how did it expand your thinking about how did it expand your thinking about how did it expand your thinking about the world how did it uh expand your the world how did it uh expand your the world how did it uh expand your thinking about thinking about thinking about you know this thing we're talking about you know this thing we're talking about you know this thing we're talking about which is humans creating another living which is humans creating another living which is humans creating another living sentient perhaps uh being sentient perhaps uh being sentient perhaps uh being I think it uh mainly do two things uh mainly do two things uh one of them just going through all their one of them just going through all their one of them just going through all their stuff after they had passed away and so stuff after they had passed away and so stuff after they had passed away and so on just drove home to me how important on just drove home to me how important on just drove home to me how important it is to ask ourselves it is to ask ourselves it is to ask ourselves why are we doing this things we do why are we doing this things we do why are we doing this things we do because it's inevitable that you look at because it's inevitable that you look at because it's inevitable that you look at some things they spent an enormous time some things they spent an enormous time some things they spent an enormous time on and you asked in hindsight would they on and you asked in hindsight would they on and you asked in hindsight would they really have spent so much time on this really have spent so much time on this really have spent so much time on this or if would they have done something or if would they have done something or if would they have done something that was more meaningful that was more meaningful that was more meaningful um so I've been looking more in my life um so I've been looking more in my life um so I've been looking more in my life now and asking you know why am I doing now and asking you know why am I doing now and asking you know why am I doing what I'm doing and I I feel it should either be something I really it should either be something I really enjoy doing or it should be something enjoy doing or it should be something enjoy doing or it should be something that I find really really meaningful that I find really really meaningful that I find really really meaningful because it helps because it helps because it helps Humanity Humanity Humanity and um and um and um if it's in none of those two categories if it's in none of those two categories if it's in none of those two categories maybe I should spend less time on it you maybe I should spend less time on it you maybe I should spend less time on it you know the other thing is dealing with know the other thing is dealing with know the other thing is dealing with death up in personal like this it's death up in personal like this it's death up in personal like this it's actually made me less afraid actually made me less afraid actually made me less afraid of of of um um um even less afraid of other people telling

even less afraid of other people telling even less afraid of other people telling me that I'm an idiot you know which me that I'm an idiot you know which me that I'm an idiot you know which happens regularly and just live my life happens regularly and just live my life happens regularly and just live my life do my thing you know do my thing you know do my thing you know um um um and um and um and um it's made it a little bit easier for me it's made it a little bit easier for me it's made it a little bit easier for me to focus on what I what I feel is really to focus on what I what I feel is really to focus on what I what I feel is really important what about fear of your own important what about fear of your own important what about fear of your own death death death has it made it more real that this is has it made it more real that this is has it made it more real that this is that this is something that happens yeah that this is something that happens yeah that this is something that happens yeah it's made it extremely real and I'm next it's made it extremely real and I'm next it's made it extremely real and I'm next next in line in our family now right next in line in our family now right next in line in our family now right it's me and my brother my younger it's me and my brother my younger it's me and my brother my younger brother but um brother but um brother but um they both handled it with such dignity they both handled it with such dignity they both handled it with such dignity it was there was a true inspiration also it was there was a true inspiration also it was there was a true inspiration also they never complained about things and they never complained about things and they never complained about things and you know when you're old and your body you know when you're old and your body you know when you're old and your body starts falling apart it's more and more starts falling apart it's more and more starts falling apart it's more and more to complain about they looked at what to complain about they looked at what to complain about they looked at what could they still do that was meaningful could they still do that was meaningful could they still do that was meaningful and they focused on that rather than and they focused on that rather than and they focused on that rather than wasting time talking about or even thinking much talking about or even thinking much about things they were disappointed in about things they were disappointed in about things they were disappointed in I think anyone can make themselves I think anyone can make themselves I think anyone can make themselves depressed if they start their morning by depressed if they start their morning by depressed if they start their morning by making a list of grievances making a list of grievances making a list of grievances whereas if you start your day and when whereas if you start your day and when whereas if you start your day and when the little meditation and just the the little meditation and just the the little meditation and just the things you're grateful for you you things you're grateful for you you things you're grateful for you you basically choose to be a happy person basically choose to be a happy person basically choose to be a happy person because you only have a finite number of because you only have a finite number of because you only have a finite number of days you should spend them Make It Count days you should spend them Make It Count days you should spend them Make It Count being grateful yeah well you do happen to be working on a well you do happen to be working on a thing which seems to have a potentially

thing which seems to have a potentially thing which seems to have a potentially some of the greatest impact on human some of the greatest impact on human some of the greatest impact on human civilization of anything humans have civilization of anything humans have civilization of anything humans have ever created which is artificial ever created which is artificial ever created which is artificial intelligence this is on the both intelligence this is on the both intelligence this is on the both detailed technical level and in the high detailed technical level and in the high detailed technical level and in the high philosophical level you work on so philosophical level you work on so philosophical level you work on so you've mentioned to me that there's an you've mentioned to me that there's an you've mentioned to me that there's an open letter open letter open letter that you're working on it's actually uh that you're working on it's actually uh that you're working on it's actually uh going live in a few hours so I've been going live in a few hours so I've been going live in a few hours so I've been having late nights and early mornings having late nights and early mornings having late nights and early mornings it's been very exciting actually I in it's been very exciting actually I in it's been very exciting actually I in short short short I have you seen uh don't look up I have you seen uh don't look up I have you seen uh don't look up the film the film the film yes yes I don't want to be the movie yes yes I don't want to be the movie yes yes I don't want to be the movie spoiler for anyone watching this who spoiler for anyone watching this who spoiler for anyone watching this who hasn't seen it but if you're watching hasn't seen it but if you're watching hasn't seen it but if you're watching this you haven't seen it watch it this you haven't seen it watch it this you haven't seen it watch it because we are actually acting out it's because we are actually acting out it's because we are actually acting out it's it's life imitating art humanity is it's life imitating art humanity is it's life imitating art humanity is doing exactly that right now except doing exactly that right now except doing exactly that right now except it's an asteroid that we are building it's an asteroid that we are building it's an asteroid that we are building ourselves ourselves ourselves almost nobody is talking about it almost nobody is talking about it almost nobody is talking about it people are squabbling across the planet people are squabbling across the planet people are squabbling across the planet about all sorts of things which seem about all sorts of things which seem about all sorts of things which seem very minor compared to the asteroid very minor compared to the asteroid very minor compared to the asteroid that's about to hit us right uh most that's about to hit us right uh most that's about to hit us right uh most politicians don't even have their radar politicians don't even have their radar politicians don't even have their radar this on the radar they think maybe in this on the radar they think maybe in this on the radar they think maybe in 100 years or whatever 100 years or whatever 100 years or whatever right now right now right now we're at a fork on the road this is the we're at a fork on the road this is the we're at a fork on the road this is the most important um Fork the humanity has most important um Fork the humanity has most important um Fork the humanity has reached in its over a hundred thousand reached in its over a hundred thousand reached in its over a hundred thousand years on this planet we're building years on this planet we're building years on this planet we're building effectively a new species that's smarter effectively a new species that's smarter effectively a new species that's smarter than us than us than us it doesn't look so much like a species it doesn't look so much like a species it doesn't look so much like a species yet because it's mostly not embodied in

yet because it's mostly not embodied in yet because it's mostly not embodied in robots but um robots but um robots but um that's a technicality which will soon be that's a technicality which will soon be that's a technicality which will soon be changed and and this arrival of changed and and this arrival of changed and and this arrival of artificial general intelligence that can artificial general intelligence that can artificial general intelligence that can do all our jobs as well as us and do all our jobs as well as us and do all our jobs as well as us and probably shortly thereafter super probably shortly thereafter super probably shortly thereafter super intelligence which greatly exceeds our intelligence which greatly exceeds our intelligence which greatly exceeds our cognitive abilities it's going to either cognitive abilities it's going to either cognitive abilities it's going to either be the the best thing ever to happen be the the best thing ever to happen be the the best thing ever to happen Humanity or the worst I'm really quite Humanity or the worst I'm really quite Humanity or the worst I'm really quite confident that there is confident that there is confident that there is not that much Middle Ground there but it not that much Middle Ground there but it not that much Middle Ground there but it would be fundamentally transformative would be fundamentally transformative would be fundamentally transformative to human civilization of course utterly to human civilization of course utterly to human civilization of course utterly and totally you know again we branded and totally you know again we branded and totally you know again we branded ourselves as Homo sapiens because it ourselves as Homo sapiens because it ourselves as Homo sapiens because it seemed like the basic thing where the seemed like the basic thing where the seemed like the basic thing where the King of the castle on this planet were King of the castle on this planet were King of the castle on this planet were the Smart Ones if we can control the Smart Ones if we can control the Smart Ones if we can control everything else everything else everything else this could very easily change we're this could very easily change we're this could very easily change we're certainly not going to be the smartest certainly not going to be the smartest certainly not going to be the smartest on the planet for very long if AI unless on the planet for very long if AI unless on the planet for very long if AI unless AI progress just Falls and we can talk AI progress just Falls and we can talk AI progress just Falls and we can talk more about why I I think that's true more about why I I think that's true more about why I I think that's true because it's it's controversial because it's it's controversial because it's it's controversial and and then we can also talk about reasons we might think it's gonna be the reasons we might think it's gonna be the best thing ever and the reason you think best thing ever and the reason you think best thing ever and the reason you think it's going to be the end of humanity it's going to be the end of humanity it's going to be the end of humanity which is of course super controversial which is of course super controversial which is of course super controversial but but but what I think we can anyone who's working what I think we can anyone who's working what I think we can anyone who's working on uh Advanced AI on uh Advanced AI on uh Advanced AI can agree on is it's it's much like the can agree on is it's it's much like the can agree on is it's it's much like the film don't look up and that film don't look up and that film don't look up and that it's just really comical how little it's just really comical how little it's just really comical how little serious public debate there is about it

serious public debate there is about it serious public debate there is about it given how huge it is so what we're talking about is the so what we're talking about is the development of currently things like development of currently things like development of currently things like gpt4 gpt4 gpt4 and the signs it's showing of uh rapid and the signs it's showing of uh rapid and the signs it's showing of uh rapid Improvement that may in the near term Improvement that may in the near term Improvement that may in the near term lead to development of super intelligent lead to development of super intelligent lead to development of super intelligent AGI AI General AI systems and what kind AGI AI General AI systems and what kind AGI AI General AI systems and what kind of impact that has on society exactly of impact that has on society exactly of impact that has on society exactly when that thing is achieves General when that thing is achieves General when that thing is achieves General human level intelligence and then beyond human level intelligence and then beyond human level intelligence and then beyond that General superhuman level that General superhuman level that General superhuman level intelligence intelligence intelligence there's a lot of questions to explore there's a lot of questions to explore there's a lot of questions to explore here so one you mentioned halt is that here so one you mentioned halt is that here so one you mentioned halt is that uh the content of the letter is to uh the content of the letter is to uh the content of the letter is to suggest that maybe we should pause the suggest that maybe we should pause the suggest that maybe we should pause the development of these systems exactly so development of these systems exactly so development of these systems exactly so this is very controversial this is very controversial this is very controversial from from from when we talked the first time we talked when we talked the first time we talked when we talked the first time we talked about how I was involved in starting the about how I was involved in starting the about how I was involved in starting the future Life Institute and we worked very future Life Institute and we worked very future Life Institute and we worked very hard on 2014-2015 was the mainstream AI hard on 2014-2015 was the mainstream AI hard on 2014-2015 was the mainstream AI safety safety safety the idea that there even could be risks the idea that there even could be risks the idea that there even could be risks and that you could do things about them and that you could do things about them and that you could do things about them before then a lot of people thought it before then a lot of people thought it before then a lot of people thought it was just really kooky to even talk about was just really kooky to even talk about was just really kooky to even talk about it and a lot of AI researchers felt it and a lot of AI researchers felt it and a lot of AI researchers felt worried that this was too flaky and worried that this was too flaky and worried that this was too flaky and could be bad for funding and that the could be bad for funding and that the could be bad for funding and that the people had talked about it or just not people had talked about it or just not people had talked about it or just not didn't understand AI didn't understand AI didn't understand AI I'm very very happy with I'm very very happy with I'm very very happy with how that's gone in that now you know

how that's gone in that now you know how that's gone in that now you know just completely mainstream you go on any just completely mainstream you go on any just completely mainstream you go on any AI conference and people talk about AI AI conference and people talk about AI AI conference and people talk about AI safety and it's a nerdy technical field safety and it's a nerdy technical field safety and it's a nerdy technical field full of equations and simula and blah full of equations and simula and blah full of equations and simula and blah blah yes blah yes blah yes um um um as it should be uh as it should be uh as it should be uh but there's this other thing which has but there's this other thing which has but there's this other thing which has been quite taboo up until now been quite taboo up until now been quite taboo up until now calling for slowdown so what calling for slowdown so what calling for slowdown so what we've been constantly been saying we've been constantly been saying we've been constantly been saying including myself I've been biting my including myself I've been biting my including myself I've been biting my tongue a lot you know is that you know tongue a lot you know is that you know tongue a lot you know is that you know we we don't need to slow down AI we we don't need to slow down AI we we don't need to slow down AI development we just need to win this development we just need to win this development we just need to win this race the wisdom race between the growing race the wisdom race between the growing race the wisdom race between the growing power of the AI and the growing wisdom power of the AI and the growing wisdom power of the AI and the growing wisdom with which we manage it and rather than with which we manage it and rather than with which we manage it and rather than trying to slow down AI let's just try to trying to slow down AI let's just try to trying to slow down AI let's just try to accelerate the wisdom do all this accelerate the wisdom do all this accelerate the wisdom do all this technical work to figure out how you can technical work to figure out how you can technical work to figure out how you can actually ensure that your powerful AI is actually ensure that your powerful AI is actually ensure that your powerful AI is going to do what you wanted to do and going to do what you wanted to do and going to do what you wanted to do and have Society adapt also have Society adapt also have Society adapt also with um incentives and regulations so with um incentives and regulations so with um incentives and regulations so that these things get put to good use that these things get put to good use that these things get put to good use um sadly that um sadly that um sadly that didn't pan out didn't pan out didn't pan out the progress on technical Ai and the progress on technical Ai and the progress on technical Ai and capabilities has gone a lot faster than capabilities has gone a lot faster than capabilities has gone a lot faster than than many people thought than many people thought than many people thought back when we started this in 2014 turned back when we started this in 2014 turned back when we started this in 2014 turned out to be easier to build really out to be easier to build really out to be easier to build really Advanced AI than we thought Advanced AI than we thought Advanced AI than we thought um um um and on the other side it's gone much and on the other side it's gone much and on the other side it's gone much slower than we hoped with getting

slower than we hoped with getting slower than we hoped with getting um um um policy makers and others to actually policy makers and others to actually policy makers and others to actually put them incentives in place to to make put them incentives in place to to make put them incentives in place to to make steer this in the in the good directions steer this in the in the good directions steer this in the in the good directions we can maybe we should unpack it and we can maybe we should unpack it and we can maybe we should unpack it and talk a little bit about each so yeah why talk a little bit about each so yeah why talk a little bit about each so yeah why did it go faster than we than a lot of did it go faster than we than a lot of did it go faster than we than a lot of people thought them people thought them people thought them in hindsight it's exactly like building in hindsight it's exactly like building in hindsight it's exactly like building um um um flying machines flying machines flying machines people spent a lot of time wondering people spent a lot of time wondering people spent a lot of time wondering about how the birds fly you know and about how the birds fly you know and about how the birds fly you know and that turned out to be really hard have that turned out to be really hard have that turned out to be really hard have you seen the Ted talk with a flying bird you seen the Ted talk with a flying bird you seen the Ted talk with a flying bird like a flying robotic Bird yeah it flies like a flying robotic Bird yeah it flies like a flying robotic Bird yeah it flies around the audience but it took a around the audience but it took a around the audience but it took a hundred years longer to figure out how hundred years longer to figure out how hundred years longer to figure out how to do that than for the Wright brothers to do that than for the Wright brothers to do that than for the Wright brothers to build the first airplane because it to build the first airplane because it to build the first airplane because it turned out there was a much easier way turned out there was a much easier way turned out there was a much easier way to fly to fly to fly and evolution picked a more complicated and evolution picked a more complicated and evolution picked a more complicated one because it had its hands tied it one because it had its hands tied it one because it had its hands tied it could only build a machine that could could only build a machine that could could only build a machine that could assemble itself which the Wright assemble itself which the Wright assemble itself which the Wright brothers didn't care about they can only brothers didn't care about they can only brothers didn't care about they can only build a machine they'll use only the build a machine they'll use only the build a machine they'll use only the most common atoms in the periodic table most common atoms in the periodic table most common atoms in the periodic table Wright brothers didn't care about that Wright brothers didn't care about that Wright brothers didn't care about that they could use steel they could use steel they could use steel iron atoms and it had to be able to iron atoms and it had to be able to iron atoms and it had to be able to repair itself and it also had to be repair itself and it also had to be repair itself and it also had to be incredibly fuel efficient you know incredibly fuel efficient you know incredibly fuel efficient you know a lot of birds use less than half the a lot of birds use less than half the a lot of birds use less than half the fuel of a remote control plane that's fuel of a remote control plane that's fuel of a remote control plane that's flying the same distance flying the same distance flying the same distance for humans let's throw a little more put for humans let's throw a little more put for humans let's throw a little more put a little more fuel in a roof there you a little more fuel in a roof there you a little more fuel in a roof there you go 100 years earlier go 100 years earlier go 100 years earlier that's exactly what's happening now with that's exactly what's happening now with that's exactly what's happening now with these large language models these large language models these large language models the brain is incredibly complicated the brain is incredibly complicated the brain is incredibly complicated many people made the mistake you're

many people made the mistake you're many people made the mistake you're thinking we had to figure out how the thinking we had to figure out how the thinking we had to figure out how the brain does human level AI first before brain does human level AI first before brain does human level AI first before we could build in the machine we could build in the machine we could build in the machine that was completely wrong you can take that was completely wrong you can take that was completely wrong you can take an incredibly simple computational system called the computational system called the Transformer Network and just train it to Transformer Network and just train it to Transformer Network and just train it to do something incredibly dumb do something incredibly dumb do something incredibly dumb just read a gigantic amount of text and just read a gigantic amount of text and just read a gigantic amount of text and try to predict the next word try to predict the next word try to predict the next word and it turns out and it turns out and it turns out if you just throw a ton of compute at if you just throw a ton of compute at if you just throw a ton of compute at that and a ton of data it gets to be that and a ton of data it gets to be that and a ton of data it gets to be frighteningly good like gpt4 which I've frighteningly good like gpt4 which I've frighteningly good like gpt4 which I've been playing with so much since it came been playing with so much since it came been playing with so much since it came out right out right out right and um and um and um there's still some debate about whether there's still some debate about whether there's still some debate about whether that can get you all the way to full that can get you all the way to full that can get you all the way to full human level or not human level or not human level or not but uh yeah we can come back to the but uh yeah we can come back to the but uh yeah we can come back to the details of that and how you might get details of that and how you might get details of that and how you might get the human level AI even if the human level AI even if the human level AI even if a large language models don't a large language models don't a large language models don't can you briefly if it's just a small can you briefly if it's just a small can you briefly if it's just a small tangent comment on your feelings about tangent comment on your feelings about tangent comment on your feelings about gpt4 so just that you're impressed by gpt4 so just that you're impressed by gpt4 so just that you're impressed by this rate of progress but where where is this rate of progress but where where is this rate of progress but where where is it can gpt4 reason it can gpt4 reason it can gpt4 reason what are like the intuitions what are what are like the intuitions what are what are like the intuitions what are human interpretable words you can assign human interpretable words you can assign human interpretable words you can assign to the capabilities of gpt4 that makes to the capabilities of gpt4 that makes to the capabilities of gpt4 that makes you so damn impressed with it I'm both you so damn impressed with it I'm both you so damn impressed with it I'm both very excited about it and terrified very excited about it and terrified very excited about it and terrified interesting mixture of promotions all interesting mixture of promotions all interesting mixture of promotions all the best things in life include those the best things in life include those the best things in life include those two somehow yeah I can absolutely reason two somehow yeah I can absolutely reason two somehow yeah I can absolutely reason anyone who hasn't played with it I anyone who hasn't played with it I anyone who hasn't played with it I highly recommend doing that before

highly recommend doing that before highly recommend doing that before dissing it dissing it dissing it it can do quite quite remarkable it can do quite quite remarkable it can do quite quite remarkable reasoning and reasoning and reasoning and I've had to do a lot of things which I I've had to do a lot of things which I I've had to do a lot of things which I realized I couldn't do that myself that realized I couldn't do that myself that realized I couldn't do that myself that well even and and obviously does it well even and and obviously does it well even and and obviously does it dramatically faster than we do too when dramatically faster than we do too when dramatically faster than we do too when you watch it type you watch it type you watch it type and it's doing that while servicing a and it's doing that while servicing a and it's doing that while servicing a massive number of other humans at the massive number of other humans at the massive number of other humans at the same time at the same time it cannot same time at the same time it cannot same time at the same time it cannot reason reason reason as well as a human can on some tasks as well as a human can on some tasks as well as a human can on some tasks just because it's obviously a limitation just because it's obviously a limitation just because it's obviously a limitation from its architecture you know we have from its architecture you know we have from its architecture you know we have in our heads what in geekspeak is called in our heads what in geekspeak is called in our heads what in geekspeak is called the recurrent neural network there are the recurrent neural network there are the recurrent neural network there are Loops information can go from this Loops information can go from this Loops information can go from this neuron the base neuron to this neuron neuron the base neuron to this neuron neuron the base neuron to this neuron and then back to this one you can like and then back to this one you can like and then back to this one you can like ruminate on something for a while you ruminate on something for a while you ruminate on something for a while you can self-reflect a lot uh these large can self-reflect a lot uh these large can self-reflect a lot uh these large language models that are they cannot language models that are they cannot language models that are they cannot like gpt4 it's it's a so-called like gpt4 it's it's a so-called like gpt4 it's it's a so-called Transformer where it's just like a Transformer where it's just like a Transformer where it's just like a one-way Street of information basically one-way Street of information basically one-way Street of information basically and geekspeak it's called the feed and geekspeak it's called the feed and geekspeak it's called the feed forward neural network forward neural network forward neural network and it's only so deep so it can only do and it's only so deep so it can only do and it's only so deep so it can only do logic that's that many steps and that logic that's that many steps and that logic that's that many steps and that deep and it's not deep and it's not deep and it's not and you can so you can create problems and you can so you can create problems and you can so you can create problems which will fail to solve you know for which will fail to solve you know for which will fail to solve you know for that reason but the fact that it can do so amazing but the fact that it can do so amazing things with this incredible simple things with this incredible simple things with this incredible simple architecture already it's quite stunning architecture already it's quite stunning architecture already it's quite stunning and and what we see in my lab at MIT and and what we see in my lab at MIT and and what we see in my lab at MIT when we look inside when we look inside when we look inside large language models to try to figure large language models to try to figure large language models to try to figure out how they're doing it that's the key

out how they're doing it that's the key out how they're doing it that's the key core focus of our research it's called core focus of our research it's called core focus of our research it's called um mechanistic interpretability in geek um mechanistic interpretability in geek um mechanistic interpretability in geek speak you know you have this machine it speak you know you have this machine it speak you know you have this machine it does something smart you try to reverse does something smart you try to reverse does something smart you try to reverse reverse engineer see how does it do it reverse engineer see how does it do it reverse engineer see how does it do it are you think of it also as artificial are you think of it also as artificial are you think of it also as artificial Neuroscience that's exactly what Neuroscience that's exactly what Neuroscience that's exactly what neuroscientists do with actual brains neuroscientists do with actual brains neuroscientists do with actual brains but here you have the advantage that you but here you have the advantage that you but here you have the advantage that you can you don't have to worry about can you don't have to worry about can you don't have to worry about measurement errors you can see what measurement errors you can see what measurement errors you can see what every neuron is doing all the time and every neuron is doing all the time and every neuron is doing all the time and and a recurrent thing we see again and and a recurrent thing we see again and and a recurrent thing we see again and again again again there's been a number of beautiful there's been a number of beautiful there's been a number of beautiful papers quite recently by papers quite recently by papers quite recently by by a lot of researchers some of them by a lot of researchers some of them by a lot of researchers some of them here I am in this area is where when here I am in this area is where when here I am in this area is where when they figure out how something is done they figure out how something is done they figure out how something is done you can say oh man that's such a dumb you can say oh man that's such a dumb you can say oh man that's such a dumb way of doing it and you immediately see way of doing it and you immediately see way of doing it and you immediately see how it can be improved like for example how it can be improved like for example how it can be improved like for example there was a beautiful paper recently there was a beautiful paper recently there was a beautiful paper recently where they figured out how a large where they figured out how a large where they figured out how a large language model stores certain facts like language model stores certain facts like language model stores certain facts like Eiffel Towers in Paris Eiffel Towers in Paris Eiffel Towers in Paris and they figured out exactly how it's and they figured out exactly how it's and they figured out exactly how it's stored and where the proof that they stored and where the proof that they stored and where the proof that they understood it was they could edit it understood it was they could edit it understood it was they could edit it they changed some of the synapses in it they changed some of the synapses in it they changed some of the synapses in it and then they asked it where's the and then they asked it where's the and then they asked it where's the Eiffel Tower and it said it's in Rome Eiffel Tower and it said it's in Rome Eiffel Tower and it said it's in Rome and then they asked you know how do you and then they asked you know how do you and then they asked you know how do you get there oh how do you get there from get there oh how do you get there from get there oh how do you get there from Germany oh you take this train and to Germany oh you take this train and to Germany oh you take this train and to Roma Termini train station and this and Roma Termini train station and this and Roma Termini train station and this and that and what might you see if you're in that and what might you see if you're in that and what might you see if you're in front of it oh you might see the front of it oh you might see the front of it oh you might see the Coliseum so they had edit it so they Coliseum so they had edit it so they Coliseum so they had edit it so they literally moved it to Rome but it the literally moved it to Rome but it the literally moved it to Rome but it the way it's storing this information it's way it's storing this information it's way it's storing this information it's incredibly dumb for for any fellow nerds

incredibly dumb for for any fellow nerds incredibly dumb for for any fellow nerds listening to this there was a big Matrix listening to this there was a big Matrix listening to this there was a big Matrix and a and a and a and roughly speaking there are certain and roughly speaking there are certain and roughly speaking there are certain row and column vectors which encode row and column vectors which encode row and column vectors which encode these things and the they correspond these things and the they correspond these things and the they correspond very highly related principal components very highly related principal components very highly related principal components and it will be much more efficient for and it will be much more efficient for and it will be much more efficient for sparse Matrix just store it in the sparse Matrix just store it in the sparse Matrix just store it in the database you know and and but and database you know and and but and database you know and and but and everything so far we've figured out how everything so far we've figured out how everything so far we've figured out how these things do our ways where you can these things do our ways where you can these things do our ways where you can see they can easily be improved and the see they can easily be improved and the see they can easily be improved and the fact that this particular architecture fact that this particular architecture fact that this particular architecture has some roadblocks built into it is in has some roadblocks built into it is in has some roadblocks built into it is in no way going to prevent um craft the no way going to prevent um craft the no way going to prevent um craft the researchers from quickly finding researchers from quickly finding researchers from quickly finding workarounds and making workarounds and making workarounds and making other kinds of architectures other kinds of architectures other kinds of architectures sort of go all the way so so it's um in sort of go all the way so so it's um in sort of go all the way so so it's um in short it's turned out to be a lot short it's turned out to be a lot short it's turned out to be a lot easier to build human close to human easier to build human close to human easier to build human close to human intelligence than we thought then that intelligence than we thought then that intelligence than we thought then that means our Runway is a species that get our together has shortened get our together has shortened and it seems like the scary thing about and it seems like the scary thing about and it seems like the scary thing about the effectiveness of large language the effectiveness of large language the effectiveness of large language models uh so Sam Altman every models uh so Sam Altman every models uh so Sam Altman every conversation with conversation with conversation with and and and he really showed that the leap from gpt3 he really showed that the leap from gpt3 he really showed that the leap from gpt3 to gpt4 has to do with just a bunch of to gpt4 has to do with just a bunch of to gpt4 has to do with just a bunch of hacks hacks hacks a bunch of a bunch of a bunch of uh little Explorations but with the uh little Explorations but with the uh little Explorations but with the smart researchers doing a few little smart researchers doing a few little smart researchers doing a few little fixes here and there it's not some fixes here and there it's not some fixes here and there it's not some fundamental leap and transformation in fundamental leap and transformation in fundamental leap and transformation in the architecture and more data and more

the architecture and more data and more the architecture and more data and more compute and more data and compute but he compute and more data and compute but he compute and more data and compute but he said the big leaps has to do with not said the big leaps has to do with not said the big leaps has to do with not the data in the compute but just the data in the compute but just the data in the compute but just learning this new discipline just like learning this new discipline just like learning this new discipline just like you said so researchers are going to you said so researchers are going to you said so researchers are going to look at these architectures and there look at these architectures and there look at these architectures and there might be big leaps where you realize might be big leaps where you realize might be big leaps where you realize wait why are we doing this in this dumb wait why are we doing this in this dumb wait why are we doing this in this dumb way yeah and all of a sudden this model way yeah and all of a sudden this model way yeah and all of a sudden this model is 10x smarter yeah and that that can is 10x smarter yeah and that that can is 10x smarter yeah and that that can happen on any one day on anyone Tuesday happen on any one day on anyone Tuesday happen on any one day on anyone Tuesday or Wednesday afternoon and then all of a or Wednesday afternoon and then all of a or Wednesday afternoon and then all of a sudden you have a system that's 10x sudden you have a system that's 10x sudden you have a system that's 10x smarter smarter smarter um it seems like it's such a new um it seems like it's such a new um it seems like it's such a new discipline it's such a new like we discipline it's such a new like we discipline it's such a new like we understand so little about why this understand so little about why this understand so little about why this thing works so damn well that uh the thing works so damn well that uh the thing works so damn well that uh the linear Improvement of compute or linear Improvement of compute or linear Improvement of compute or exponential but the steady Improvement exponential but the steady Improvement exponential but the steady Improvement of compute steady Improvement of the of compute steady Improvement of the of compute steady Improvement of the data may not be the thing that even data may not be the thing that even data may not be the thing that even leads to the next leap it could be a leads to the next leap it could be a leads to the next leap it could be a surprise little hack that improves surprise little hack that improves surprise little hack that improves everything for a lot of little leaps everything for a lot of little leaps everything for a lot of little leaps here and there because here and there because here and there because because so much of this is out in the because so much of this is out in the because so much of this is out in the open also open also open also so many smart people are looking at this so many smart people are looking at this so many smart people are looking at this and trying to figure out little leaps and trying to figure out little leaps and trying to figure out little leaps here and there and uh it becomes this here and there and uh it becomes this here and there and uh it becomes this sort of collective race where if a lot sort of collective race where if a lot sort of collective race where if a lot of people feel if I don't take the leap of people feel if I don't take the leap of people feel if I don't take the leap someone else with and this is actually someone else with and this is actually someone else with and this is actually very crucial for for the other part but very crucial for for the other part but very crucial for for the other part but why do we want to slow this down so why do we want to slow this down so why do we want to slow this down so again what this open letter is calling again what this open letter is calling again what this open letter is calling for is just pausing for is just pausing for is just pausing all training all training all training of uh of uh of uh systems that are more powerful than gpt4 systems that are more powerful than gpt4 systems that are more powerful than gpt4 for six months for six months for six months let's give a chance let's give a chance let's give a chance for the labs to coordinate a bit on for the labs to coordinate a bit on for the labs to coordinate a bit on safety and for society to adapt give the

safety and for society to adapt give the safety and for society to adapt give the right incentives to the labs because I right incentives to the labs because I right incentives to the labs because I you know you've interviewed a lot of you know you've interviewed a lot of you know you've interviewed a lot of these these these people who lead these labs and you know people who lead these labs and you know people who lead these labs and you know just as well as I do that they're good just as well as I do that they're good just as well as I do that they're good people they're idealistic people they're people they're idealistic people they're people they're idealistic people they're doing this doing this doing this first and foremost because they believe first and foremost because they believe first and foremost because they believe that AI has a huge potential to help that AI has a huge potential to help that AI has a huge potential to help humanity and uh humanity and uh humanity and uh but at the same time they are trapped in but at the same time they are trapped in but at the same time they are trapped in this horrible race to the bottom have you read meditations on malloc have you read meditations on malloc by Scott Alexander yes yeah it's a by Scott Alexander yes yeah it's a by Scott Alexander yes yeah it's a beautiful essay on this poem by Ginsburg beautiful essay on this poem by Ginsburg beautiful essay on this poem by Ginsburg where he interprets it as being about where he interprets it as being about where he interprets it as being about this monster it's this game theory monster that that it's this game theory monster that that pits people into against each other in pits people into against each other in pits people into against each other in this they race the bottom where this they race the bottom where this they race the bottom where everybody ultimately loses the edit the everybody ultimately loses the edit the everybody ultimately loses the edit the evil thing about this monster is even evil thing about this monster is even evil thing about this monster is even though everybody sees it and understands though everybody sees it and understands though everybody sees it and understands they still can't get out of the race they still can't get out of the race they still can't get out of the race right right right most a good fraction of all the bad most a good fraction of all the bad most a good fraction of all the bad things that we humans do are caused by things that we humans do are caused by things that we humans do are caused by moloch and I I like uh Scott Alexander's moloch and I I like uh Scott Alexander's moloch and I I like uh Scott Alexander's um um um naming of the monster so we can we naming of the monster so we can we naming of the monster so we can we humans can think of it as an f a thing humans can think of it as an f a thing humans can think of it as an f a thing if you look at why do we have if you look at why do we have if you look at why do we have overfishing why do we have more overfishing why do we have more overfishing why do we have more generally the tragedy of the commons why generally the tragedy of the commons why generally the tragedy of the commons why is it that um is it that um is it that um I don't know if you've had her on your I don't know if you've had her on your I don't know if you've had her on your podcast yeah she's become a friend yeah podcast yeah she's become a friend yeah podcast yeah she's become a friend yeah great she made this awesome point great she made this awesome point great she made this awesome point recently that beauty filters that a lot

recently that beauty filters that a lot recently that beauty filters that a lot of female of female of female influencers feel pressured to use or influencers feel pressured to use or influencers feel pressured to use or exactly malloc in action again first exactly malloc in action again first exactly malloc in action again first nobody was using them nobody was using them nobody was using them and people saw them just the way they and people saw them just the way they and people saw them just the way they were and then some of them started using were and then some of them started using were and then some of them started using it it it and becoming ever more Plastic Fantastic and becoming ever more Plastic Fantastic and becoming ever more Plastic Fantastic and then the other ones they weren't and then the other ones they weren't and then the other ones they weren't using he started to realize that using he started to realize that using he started to realize that if they want to just keep their if they want to just keep their if they want to just keep their their market share they have to start their market share they have to start their market share they have to start using it too using it too using it too and that and then you're in a situation and that and then you're in a situation and that and then you're in a situation where they're all using it where they're all using it where they're all using it and and none of them has any more market and and none of them has any more market and and none of them has any more market share or less than before so nobody share or less than before so nobody share or less than before so nobody gained anything everybody lost gained anything everybody lost gained anything everybody lost and they have to keep becoming ever more and they have to keep becoming ever more and they have to keep becoming ever more Plastic Fantastic also right Plastic Fantastic also right Plastic Fantastic also right and uh and uh and uh but nobody can go back to the old way but nobody can go back to the old way but nobody can go back to the old way because it's just because it's just because it's just too costly right the malloc is too costly right the malloc is too costly right the malloc is everywhere everywhere everywhere and um and um and um molok is not a new arrival on on the molok is not a new arrival on on the molok is not a new arrival on on the scene either we humans have developed a scene either we humans have developed a scene either we humans have developed a lot of collaboration mechanisms to help lot of collaboration mechanisms to help lot of collaboration mechanisms to help us fight back against Malik through us fight back against Malik through us fight back against Malik through various kinds of constructive various kinds of constructive various kinds of constructive collaboration the Soviet Union and the collaboration the Soviet Union and the collaboration the Soviet Union and the United States did sign the number of our United States did sign the number of our United States did sign the number of our Arms Control treaties Arms Control treaties Arms Control treaties against moloch who is trying to stoke against moloch who is trying to stoke against moloch who is trying to stoke them into them into them into unnecessarily risky nuclear arms races unnecessarily risky nuclear arms races unnecessarily risky nuclear arms races Etc et cetera and this is exactly what's Etc et cetera and this is exactly what's Etc et cetera and this is exactly what's happening on the AI front this time happening on the AI front this time happening on the AI front this time it's a little bit geopolitics but it's it's a little bit geopolitics but it's it's a little bit geopolitics but it's mostly money where there's just so much mostly money where there's just so much mostly money where there's just so much commercial pressure you know if you take

commercial pressure you know if you take commercial pressure you know if you take any of these any of these any of these leaders of the top tech companies leaders of the top tech companies leaders of the top tech companies and if they just say you know this is and if they just say you know this is and if they just say you know this is too risky I want to pause too risky I want to pause too risky I want to pause for six months they're going to get a for six months they're going to get a for six months they're going to get a lot of pressure from shareholders and lot of pressure from shareholders and lot of pressure from shareholders and others others others we're like well you know if you pause we're like well you know if you pause we're like well you know if you pause but those guys don't pause but those guys don't pause but those guys don't pause we're we're we're if you don't want to get our lunch eaten if you don't want to get our lunch eaten if you don't want to get our lunch eaten yeah and shareholders even have the yeah and shareholders even have the yeah and shareholders even have the power to replace the the executives in power to replace the the executives in power to replace the the executives in the worst case right so the worst case right so the worst case right so we did this open letter because we want we did this open letter because we want we did this open letter because we want to help these idealistic Tech Executives to help these idealistic Tech Executives to help these idealistic Tech Executives to do to do to do what their heart tells them by providing what their heart tells them by providing what their heart tells them by providing enough public pressure on the whole enough public pressure on the whole enough public pressure on the whole sector sector sector just pause so they can all pause just pause so they can all pause just pause so they can all pause in a coordinated fashion and I think in a coordinated fashion and I think in a coordinated fashion and I think without the public pressure none of them without the public pressure none of them without the public pressure none of them can do it alone can do it alone can do it alone push back against their shareholders no push back against their shareholders no push back against their shareholders no matter how good-hearted they are because matter how good-hearted they are because matter how good-hearted they are because Malik is a really powerful foe Malik is a really powerful foe Malik is a really powerful foe so the idea so the idea so the idea is to is to is to for the major developers of AI systems for the major developers of AI systems for the major developers of AI systems like this so we're talking about like this so we're talking about like this so we're talking about Microsoft Google Microsoft Google Microsoft Google meta meta meta and anyone else well open AI is very and anyone else well open AI is very and anyone else well open AI is very close with Microsoft and there are close with Microsoft and there are close with Microsoft and there are plenty of smaller players and throw for plenty of smaller players and throw for plenty of smaller players and throw for example anthropic which is very example anthropic which is very example anthropic which is very impressive there's conjecture there's impressive there's conjecture there's impressive there's conjecture there's many many players I don't want to make a many many players I don't want to make a many many players I don't want to make a long list so leave anyone out and long list so leave anyone out and long list so leave anyone out and um

um um for that reason it's so important that for that reason it's so important that for that reason it's so important that some coordination happens that there's some coordination happens that there's some coordination happens that there's external pressure on all of them saying external pressure on all of them saying external pressure on all of them saying you all need the Pawns because then the you all need the Pawns because then the you all need the Pawns because then the the people the researchers in they were the people the researchers in they were the people the researchers in they were these organizations the leaders who want these organizations the leaders who want these organizations the leaders who want to slow down a little bit they can say to slow down a little bit they can say to slow down a little bit they can say their shareholders you know everybody's slowing down because of this everybody's slowing down because of this pressure and and it's the right thing to pressure and and it's the right thing to pressure and and it's the right thing to do do do have you seen in history their uh have you seen in history their uh have you seen in history their uh examples what's possible to pause the examples what's possible to pause the examples what's possible to pause the model absolutely model absolutely model absolutely and even like human cloning for example and even like human cloning for example and even like human cloning for example you could make so much money on human you could make so much money on human you could make so much money on human cloning why aren't we doing it why aren't we doing it because biologists thought hard about because biologists thought hard about because biologists thought hard about this like this is way too risky we they this like this is way too risky we they this like this is way too risky we they got together well in the 70s in the got together well in the 70s in the got together well in the 70s in the cinema and decided even cinema and decided even cinema and decided even to stop a lot more stuff also just to stop a lot more stuff also just to stop a lot more stuff also just editing the human germline right editing the human germline right editing the human germline right Gene editing that goes in Gene editing that goes in Gene editing that goes in to our offspring to our offspring to our offspring and decided let's let's not do this and decided let's let's not do this and decided let's let's not do this because it's too unpredictable what it's because it's too unpredictable what it's because it's too unpredictable what it's going to lead to going to lead to going to lead to we could lose control over what happens we could lose control over what happens we could lose control over what happens to our species so they paused to our species so they paused to our species so they paused uh there was a ton of money to be made uh there was a ton of money to be made uh there was a ton of money to be made there so it's it's very doable but you there so it's it's very doable but you there so it's it's very doable but you just need you need a public awareness of just need you need a public awareness of just need you need a public awareness of the of what the risks are and the the of what the risks are and the the of what the risks are and the broader Community coming in and saying broader Community coming in and saying broader Community coming in and saying hey let's slow down and you know another hey let's slow down and you know another hey let's slow down and you know another another common pushback I get today is

another common pushback I get today is another common pushback I get today is we We Can't Stop in the west because we We Can't Stop in the west because we We Can't Stop in the west because China and in China undoubtedly they also get and in China undoubtedly they also get told we can't slow down because the West told we can't slow down because the West told we can't slow down because the West because both sides think they're the because both sides think they're the because both sides think they're the good guy yeah good guy yeah good guy yeah but look at human cloning you know but look at human cloning you know but look at human cloning you know did China Forge ahead with human cloning did China Forge ahead with human cloning did China Forge ahead with human cloning there's been exactly one human cloning there's been exactly one human cloning there's been exactly one human cloning that's actually been done that I know of that's actually been done that I know of that's actually been done that I know of it was done by a Chinese guy do you know it was done by a Chinese guy do you know it was done by a Chinese guy do you know where he is now right in jail where he is now right in jail where he is now right in jail and you know who put him there and you know who put him there and you know who put him there who Chinese government who Chinese government who Chinese government not because westerners said China look not because westerners said China look not because westerners said China look this is no the Chinese government put this is no the Chinese government put this is no the Chinese government put them there because they also felt they them there because they also felt they them there because they also felt they like control the Chinese government if like control the Chinese government if like control the Chinese government if anything maybe they are even more anything maybe they are even more anything maybe they are even more concerned about having control then the concerned about having control then the concerned about having control then the Western governments have no incentive of Western governments have no incentive of Western governments have no incentive of just losing control over where just losing control over where just losing control over where everything is going everything is going everything is going and you can also see the Ernie bot that and you can also see the Ernie bot that and you can also see the Ernie bot that was released by I believe I do recently was released by I believe I do recently was released by I believe I do recently they got a lot of pushback from the they got a lot of pushback from the they got a lot of pushback from the government and had to rein it in you government and had to rein it in you government and had to rein it in you know in a big way know in a big way know in a big way um I think once this basic message comes um I think once this basic message comes um I think once this basic message comes out that this isn't an arms race it's a out that this isn't an arms race it's a out that this isn't an arms race it's a suicide race suicide race suicide race where everybody loses if anybody's AI where everybody loses if anybody's AI where everybody loses if anybody's AI goes out of control it really changes goes out of control it really changes goes out of control it really changes the whole dynamic it it's not the whole dynamic it it's not the whole dynamic it it's not it's it's it's I'll say this again because this is this I'll say this again because this is this I'll say this again because this is this very basic point I think a lot of people very basic point I think a lot of people very basic point I think a lot of people get wrong because a lot of people get wrong because a lot of people get wrong because a lot of people dismiss the whole idea that AI can dismiss the whole idea that AI can dismiss the whole idea that AI can really get

really get really get very superhuman because they think very superhuman because they think very superhuman because they think there's something really magical about there's something really magical about there's something really magical about intelligence such that it can only exist intelligence such that it can only exist intelligence such that it can only exist in human Minds you know because they in human Minds you know because they in human Minds you know because they believe that they think it's going to believe that they think it's going to believe that they think it's going to kind of get to just more or less kind of get to just more or less kind of get to just more or less gpd4 plus plus and then that's it gpd4 plus plus and then that's it gpd4 plus plus and then that's it they don't see it as a super as a they don't see it as a super as a they don't see it as a super as a suicide race they think whoever gets suicide race they think whoever gets suicide race they think whoever gets that first they're going to control the that first they're going to control the that first they're going to control the world they're going to win world they're going to win world they're going to win that's not how it's going to be and we that's not how it's going to be and we that's not how it's going to be and we can talk again about can talk again about can talk again about the the scientific arguments from why the the scientific arguments from why the the scientific arguments from why it's not going to stop there but it's not going to stop there but it's not going to stop there but the way it's going to be is if if the way it's going to be is if if the way it's going to be is if if anybody completely loses control and you anybody completely loses control and you anybody completely loses control and you know you don't care if if know you don't care if if know you don't care if if some some if someone manages this some some if someone manages this some some if someone manages this takeover the world who really doesn't takeover the world who really doesn't takeover the world who really doesn't share your goals you probably don't share your goals you probably don't share your goals you probably don't really even care very much about what really even care very much about what really even care very much about what nationality they have you're not going nationality they have you're not going nationality they have you're not going to like it it's much worse than today to like it it's much worse than today to like it it's much worse than today uh who if it's if you live in orwellian uh who if it's if you live in orwellian uh who if it's if you live in orwellian dystopia who you what do you care who's dystopia who you what do you care who's dystopia who you what do you care who's created it right and if someone if it created it right and if someone if it created it right and if someone if it goes farther and and goes farther and and goes farther and and we just lose control even to the we just lose control even to the we just lose control even to the machines so that it's not US versus them it's US so that it's not US versus them it's US versus it versus it versus it what do you care who who created this what do you care who who created this what do you care who who created this this underlying entity which has goals this underlying entity which has goals this underlying entity which has goals different from humans ultimately and we different from humans ultimately and we different from humans ultimately and we get marginalized we get made obsolete we get marginalized we get made obsolete we get marginalized we get made obsolete we get replaced get replaced get replaced that's why what I mean when I say it's a that's why what I mean when I say it's a that's why what I mean when I say it's a suicide race you know it's um it's kind suicide race you know it's um it's kind suicide race you know it's um it's kind of like we're rushing towards this cliff of like we're rushing towards this cliff of like we're rushing towards this cliff but the closer to the cliff we get the

but the closer to the cliff we get the but the closer to the cliff we get the more Scenic the views are and the more more Scenic the views are and the more more Scenic the views are and the more money there is there and the more so we money there is there and the more so we money there is there and the more so we keep going keep going keep going but we have to also stop at some point but we have to also stop at some point but we have to also stop at some point right quit while we're ahead and uh right quit while we're ahead and uh right quit while we're ahead and uh it's um it's a suicide race which cannot be won it's a suicide race which cannot be won but the way that really benefit from it but the way that really benefit from it but the way that really benefit from it is is is to continue developing awesome AI a to continue developing awesome AI a to continue developing awesome AI a little bit slower so we make it safe little bit slower so we make it safe little bit slower so we make it safe make sure it does the things that humans make sure it does the things that humans make sure it does the things that humans want and create a condition where want and create a condition where want and create a condition where everybody wins the technology has shown everybody wins the technology has shown everybody wins the technology has shown us that you know geopolitics and and us that you know geopolitics and and us that you know geopolitics and and politics and general is not a zero-sum politics and general is not a zero-sum politics and general is not a zero-sum game at all game at all game at all so there is some rate of development so there is some rate of development so there is some rate of development that will lead that will lead that will lead us as a human species to lose control of us as a human species to lose control of us as a human species to lose control of this thing and the hope you have is that this thing and the hope you have is that this thing and the hope you have is that there's some lower level of development there's some lower level of development there's some lower level of development which will not which will not allow us which will not which will not allow us which will not which will not allow us to lose control this is an interesting to lose control this is an interesting to lose control this is an interesting thought you have about losing control so thought you have about losing control so thought you have about losing control so what if you have somebody if you're what if you have somebody if you're what if you have somebody if you're somebody like Sandra pracha or Sam somebody like Sandra pracha or Sam somebody like Sandra pracha or Sam Altman at the head of a company like Altman at the head of a company like Altman at the head of a company like this you're saying if they develop an this you're saying if they develop an this you're saying if they develop an AGI they too will lose control of it AGI they too will lose control of it AGI they too will lose control of it so no one person can maintain control no so no one person can maintain control no so no one person can maintain control no group of individuals can maintain if group of individuals can maintain if group of individuals can maintain if it's if it's created very very soon and it's if it's created very very soon and it's if it's created very very soon and as a big black box that we don't as a big black box that we don't as a big black box that we don't understand like the large language understand like the large language understand like the large language models yeah then I'm very confident models yeah then I'm very confident models yeah then I'm very confident they're going to lose control but this they're going to lose control but this they're going to lose control but this isn't just me saying you know Sam Altman isn't just me saying you know Sam Altman isn't just me saying you know Sam Altman and then Mr sabis have both said

and then Mr sabis have both said and then Mr sabis have both said themselves acknowledge that you know themselves acknowledge that you know themselves acknowledge that you know there's really great risks with this and there's really great risks with this and there's really great risks with this and they they want to slow down once they they they want to slow down once they they they want to slow down once they feel it gets scary feel it gets scary feel it gets scary it's but it's clear that they're stuck it's but it's clear that they're stuck it's but it's clear that they're stuck in this again molok is forcing them to in this again molok is forcing them to in this again molok is forcing them to go a little faster than they're go a little faster than they're go a little faster than they're comfortable with because of pressure comfortable with because of pressure comfortable with because of pressure from just commercial pressures right from just commercial pressures right from just commercial pressures right to get a bit optimistic here of course to get a bit optimistic here of course to get a bit optimistic here of course this is a problem that can be ultimately this is a problem that can be ultimately this is a problem that can be ultimately solved solved solved uh uh uh it's just to win this wisdom race it's just to win this wisdom race it's just to win this wisdom race it's clear that what we hope that was it's clear that what we hope that was it's clear that what we hope that was gonna happen hasn't happened the the gonna happen hasn't happened the the gonna happen hasn't happened the the capability progress has gone faster than capability progress has gone faster than capability progress has gone faster than a lot of people thought then and the a lot of people thought then and the a lot of people thought then and the part the progress in in the public part the progress in in the public part the progress in in the public sphere of policy making and so on has sphere of policy making and so on has sphere of policy making and so on has gone slower than we thought even the gone slower than we thought even the gone slower than we thought even the technical AI safety has gone slower a technical AI safety has gone slower a technical AI safety has gone slower a lot of the technical Safety Research was lot of the technical Safety Research was lot of the technical Safety Research was kind of banking on that um kind of banking on that um kind of banking on that um large language models and other poorly large language models and other poorly large language models and other poorly understood systems couldn't get us all understood systems couldn't get us all understood systems couldn't get us all the way that you had to build more of a the way that you had to build more of a the way that you had to build more of a kind of intelligence that you could kind of intelligence that you could kind of intelligence that you could understand maybe it could prove itself understand maybe it could prove itself understand maybe it could prove itself safe you know things like this safe you know things like this safe you know things like this and um and um and um I'm quite confident that this can be I'm quite confident that this can be I'm quite confident that this can be done um so we can reap all the benefits done um so we can reap all the benefits done um so we can reap all the benefits but we cannot do it as quickly as uh but we cannot do it as quickly as uh but we cannot do it as quickly as uh this is out of control Express train this is out of control Express train this is out of control Express train we're on now is gonna get the AGI that's we're on now is gonna get the AGI that's we're on now is gonna get the AGI that's why we need a little more time I feel why we need a little more time I feel why we need a little more time I feel is there something to be said well like is there something to be said well like is there something to be said well like Sam Allman talked about which is while Sam Allman talked about which is while Sam Allman talked about which is while we're in the pre-agi stage to release we're in the pre-agi stage to release we're in the pre-agi stage to release often and as transparently as possible

often and as transparently as possible often and as transparently as possible to learn a lot to learn a lot to learn a lot so as opposed to being extremely so as opposed to being extremely so as opposed to being extremely cautious release a lot don't uh don't cautious release a lot don't uh don't cautious release a lot don't uh don't invest in a closed development where you invest in a closed development where you invest in a closed development where you focus on AI safety while is somewhat focus on AI safety while is somewhat focus on AI safety while is somewhat dumb dumb dumb quote unquote quote unquote quote unquote uh release as often as possible and as uh release as often as possible and as uh release as often as possible and as you start to see signs of you start to see signs of you start to see signs of uh human level intelligence or uh human level intelligence or uh human level intelligence or superhuman level intelligence then you superhuman level intelligence then you superhuman level intelligence then you put a halt on it well put a halt on it well put a halt on it well what a lot of safety researchers have what a lot of safety researchers have what a lot of safety researchers have been saying for many years is the most been saying for many years is the most been saying for many years is the most dangerous things you can do with an AI dangerous things you can do with an AI dangerous things you can do with an AI is first of all teach it to write code is first of all teach it to write code is first of all teach it to write code yeah because that's the first step yeah because that's the first step yeah because that's the first step towards recursive self-improvement which towards recursive self-improvement which towards recursive self-improvement which can take it from AGI to much higher can take it from AGI to much higher can take it from AGI to much higher levels okay oops we've done that levels okay oops we've done that levels okay oops we've done that and uh another thing high risk is and uh another thing high risk is and uh another thing high risk is connected to the internet Let It Go to connected to the internet Let It Go to connected to the internet Let It Go to websites download stuff on its own and websites download stuff on its own and websites download stuff on its own and talk to people talk to people talk to people oops we've done that already you know oops we've done that already you know oops we've done that already you know Elias yukowski you said you interviewed Elias yukowski you said you interviewed Elias yukowski you said you interviewed him recently right yeah so he had this him recently right yeah so he had this him recently right yeah so he had this tweet recently which said tweet recently which said tweet recently which said gave me one of the best laughs in a gave me one of the best laughs in a gave me one of the best laughs in a while and he's like hey people used to while and he's like hey people used to while and he's like hey people used to make fun of me and say you're so stupid make fun of me and say you're so stupid make fun of me and say you're so stupid Eliezer because you're saying you're Eliezer because you're saying you're Eliezer because you're saying you're saying um saying um saying um you have to worry of obviously you have to worry of obviously you have to worry of obviously developers wants to get to like really developers wants to get to like really developers wants to get to like really strong AI first thing you're going to do strong AI first thing you're going to do strong AI first thing you're going to do is like never connect it to the internet is like never connect it to the internet is like never connect it to the internet Keep It In The Box yeah where you know Keep It In The Box yeah where you know Keep It In The Box yeah where you know where you can really study it where you can really study it where you can really study it so he had written it in the like in the so he had written it in the like in the so he had written it in the like in the meme form so it's like then yeah and

meme form so it's like then yeah and meme form so it's like then yeah and then that and then now then that and then now then that and then now let's LOL let's make a chatbot let's LOL let's make a chatbot let's LOL let's make a chatbot yeah yeah and the third thing is Stuart yeah yeah and the third thing is Stuart yeah yeah and the third thing is Stuart Russell yeah you know Russell yeah you know Russell yeah you know amazing AI researcher he had he has amazing AI researcher he had he has amazing AI researcher he had he has argued for a while that argued for a while that argued for a while that we should never teach AI anything about we should never teach AI anything about we should never teach AI anything about humans humans humans above all we should never let it learn above all we should never let it learn above all we should never let it learn about human psychology and how you about human psychology and how you about human psychology and how you manipulate humans manipulate humans manipulate humans that's the most dangerous kind of that's the most dangerous kind of that's the most dangerous kind of knowledge you can give it yeah you can knowledge you can give it yeah you can knowledge you can give it yeah you can teach it all it needs to know how to teach it all it needs to know how to teach it all it needs to know how to about how to cure cancer and stuff like about how to cure cancer and stuff like about how to cure cancer and stuff like that but don't let it read Daniel that but don't let it read Daniel that but don't let it read Daniel kahneman's book about cognitive biases kahneman's book about cognitive biases kahneman's book about cognitive biases and all that and then and all that and then and all that and then oops lol you know let's invent social oops lol you know let's invent social oops lol you know let's invent social media media media I'll recommender algorithms which do I'll recommender algorithms which do I'll recommender algorithms which do exactly that they they get so good at exactly that they they get so good at exactly that they they get so good at knowing us and pressing our buttons knowing us and pressing our buttons knowing us and pressing our buttons that we've we're starting to create a that we've we're starting to create a that we've we're starting to create a world now where we just have ever more world now where we just have ever more world now where we just have ever more hatred hatred hatred because they figured out that these because they figured out that these because they figured out that these algorithms not for out of evil but just algorithms not for out of evil but just algorithms not for out of evil but just to make money on Advertising that the to make money on Advertising that the to make money on Advertising that the best way to get more engagement best way to get more engagement best way to get more engagement the euphemism the euphemism the euphemism get people glued to their little get people glued to their little get people glued to their little rectangles right is just to make them rectangles right is just to make them rectangles right is just to make them pissed off that's really interesting pissed off that's really interesting pissed off that's really interesting that a large AI system that's doing the that a large AI system that's doing the that a large AI system that's doing the recommender system kind of task on recommender system kind of task on recommender system kind of task on social media is basically just studying social media is basically just studying social media is basically just studying human beings because it's a bunch of us human beings because it's a bunch of us human beings because it's a bunch of us rats giving it signal rats giving it signal rats giving it signal non-stop signal it'll show a thing and

non-stop signal it'll show a thing and non-stop signal it'll show a thing and it would give signal on whether we it would give signal on whether we it would give signal on whether we spread that thing we like that thing spread that thing we like that thing spread that thing we like that thing that thing increases our engagement gets that thing increases our engagement gets that thing increases our engagement gets us to return to the platform and it has us to return to the platform and it has us to return to the platform and it has that on the scale of hundreds of that on the scale of hundreds of that on the scale of hundreds of millions of people constantly so it's millions of people constantly so it's millions of people constantly so it's just learning and learning and learning just learning and learning and learning just learning and learning and learning and presumably if the param the number and presumably if the param the number and presumably if the param the number of parameters the neural network that's of parameters the neural network that's of parameters the neural network that's doing the learning and more end-to-end doing the learning and more end-to-end doing the learning and more end-to-end the learning is the learning is the learning is the more it's able is just to basically the more it's able is just to basically the more it's able is just to basically encode how to manipulate human behavior encode how to manipulate human behavior encode how to manipulate human behavior how to control humans at scale exactly how to control humans at scale exactly how to control humans at scale exactly and that is not something you think is a and that is not something you think is a and that is not something you think is a new man in his interest new man in his interest new man in his interest yes right now it's mainly letting some yes right now it's mainly letting some yes right now it's mainly letting some humans manipulate other humans for humans manipulate other humans for humans manipulate other humans for profit profit profit and Power and Power and Power which is already which is already which is already caused a lot of damage and eventually caused a lot of damage and eventually caused a lot of damage and eventually that's a sort of that's a sort of that's a sort of skill that can make ai's persuade humans skill that can make ai's persuade humans skill that can make ai's persuade humans to let them escape and whatever safety to let them escape and whatever safety to let them escape and whatever safety precautions yeah but you know there was precautions yeah but you know there was precautions yeah but you know there was a really nice article um and the New a really nice article um and the New a really nice article um and the New York Times recently by a you all know a York Times recently by a you all know a York Times recently by a you all know a Harari and and um two co-authors Harari and and um two co-authors Harari and and um two co-authors including Justin Harris from the social including Justin Harris from the social including Justin Harris from the social dilemma and dilemma and dilemma and they have this phrase in there I love they have this phrase in there I love they have this phrase in there I love Humanity's first contact with Advanced Humanity's first contact with Advanced Humanity's first contact with Advanced AI AI AI or social media or social media or social media and we lost that one and we lost that one and we lost that one we now live in a country where there's we now live in a country where there's we now live in a country where there's much more hate in the world where much more hate in the world where much more hate in the world where there's much more hate in fact there's much more hate in fact there's much more hate in fact and in our democracy that we're having and in our democracy that we're having and in our democracy that we're having this conversation then people can't even this conversation then people can't even this conversation then people can't even agree on who won the last election you agree on who won the last election you agree on who won the last election you know

know know and we humans often point fingers at and we humans often point fingers at and we humans often point fingers at other humans and say it's their fault other humans and say it's their fault other humans and say it's their fault but it's really molok and these AI but it's really molok and these AI but it's really molok and these AI algorithms we got the algorithms and then molok we got the algorithms and then molok pitted the social media companies around pitted the social media companies around pitted the social media companies around against each other so nobody could have against each other so nobody could have against each other so nobody could have a less creepy algorithm because then a less creepy algorithm because then a less creepy algorithm because then they would lose out on our Revenue to they would lose out on our Revenue to they would lose out on our Revenue to the other company is there any way to the other company is there any way to the other company is there any way to win that battle back just if we just win that battle back just if we just win that battle back just if we just Linger on this one battle that we've Linger on this one battle that we've Linger on this one battle that we've lost in terms of social media is it lost in terms of social media is it lost in terms of social media is it possible possible possible to redesign social media this very to redesign social media this very to redesign social media this very medium in which we use as a civilization medium in which we use as a civilization medium in which we use as a civilization to communicate with each other to have to communicate with each other to have to communicate with each other to have these kinds of conversations to have these kinds of conversations to have these kinds of conversations to have discourse to try to figure out how to discourse to try to figure out how to discourse to try to figure out how to solve the biggest problems in the world solve the biggest problems in the world solve the biggest problems in the world whether that's nuclear war or the whether that's nuclear war or the whether that's nuclear war or the development of AGI is is it possible development of AGI is is it possible development of AGI is is it possible uh to do social media correct I think uh to do social media correct I think uh to do social media correct I think it's not only possible but it's it's it's not only possible but it's it's it's not only possible but it's it's necessary who are we kidding that we're necessary who are we kidding that we're necessary who are we kidding that we're going to be able to solve all these going to be able to solve all these going to be able to solve all these other challenges if we can't even have a other challenges if we can't even have a other challenges if we can't even have a conversation with each other that's conversation with each other that's conversation with each other that's constructive the whole idea the key idea constructive the whole idea the key idea constructive the whole idea the key idea of democracy is that you get a bunch of of democracy is that you get a bunch of of democracy is that you get a bunch of people together people together people together and they have a real conversation the and they have a real conversation the and they have a real conversation the ones you try to Foster on this podcast ones you try to Foster on this podcast ones you try to Foster on this podcast or you respectfully listen to people you or you respectfully listen to people you or you respectfully listen to people you disagree with disagree with disagree with and you realize actually you know there and you realize actually you know there and you realize actually you know there are some things actually we some common are some things actually we some common are some things actually we some common ground we have and that's it's yeah we ground we have and that's it's yeah we ground we have and that's it's yeah we both agree let's not have a nuclear Wars both agree let's not have a nuclear Wars both agree let's not have a nuclear Wars let's not do that let's not do that let's not do that um etc etc um etc etc um etc etc we're kidding ourselves the thinking we we're kidding ourselves the thinking we we're kidding ourselves the thinking we can face the off the can face the off the can face the off the second contact with with ever more second contact with with ever more second contact with with ever more powerful AI that's happening now with powerful AI that's happening now with powerful AI that's happening now with this large language models if we can't

this large language models if we can't this large language models if we can't even even even have a functional have a functional have a functional conversation in the public space that's conversation in the public space that's conversation in the public space that's why I started to improve the news why I started to improve the news why I started to improve the news project to improve the news.org but um project to improve the news.org but um project to improve the news.org but um I I'm an optimist fundamentally in um and that there is a lot of intrinsic and that there is a lot of intrinsic goodness in in in people goodness in in in people goodness in in in people and that uh what makes the difference and that uh what makes the difference and that uh what makes the difference between someone doing good things for between someone doing good things for between someone doing good things for for Humanity and bad things is not for Humanity and bad things is not for Humanity and bad things is not some sort of fairy tale thing that this some sort of fairy tale thing that this some sort of fairy tale thing that this person was born with the evil Gene and person was born with the evil Gene and person was born with the evil Gene and this one was not born with a good Gene this one was not born with a good Gene this one was not born with a good Gene no I think it's whether we put whether no I think it's whether we put whether no I think it's whether we put whether people people people find themselves in situations that bring find themselves in situations that bring find themselves in situations that bring out the best in them or they bring out out the best in them or they bring out out the best in them or they bring out the worst in them and I feel we're the worst in them and I feel we're the worst in them and I feel we're building an internet building an internet building an internet and a society that brings out the worst but it doesn't have to be that way no it but it doesn't have to be that way no it does not it's possible to create does not it's possible to create does not it's possible to create incentives and also create incentives incentives and also create incentives incentives and also create incentives that make money they both make money and that make money they both make money and that make money they both make money and bring out the best in people I mean in bring out the best in people I mean in bring out the best in people I mean in the long term it's not a good investment the long term it's not a good investment the long term it's not a good investment for anyone you know to have a nuclear for anyone you know to have a nuclear for anyone you know to have a nuclear war for example war for example war for example and you know is it a good investment for and you know is it a good investment for and you know is it a good investment for Humanity if we just ultimately replace Humanity if we just ultimately replace Humanity if we just ultimately replace all humans by machines and then we're so all humans by machines and then we're so all humans by machines and then we're so obsolete that eventually the obsolete that eventually the obsolete that eventually the there are no humans left there are no humans left there are no humans left well it depends against how you do the well it depends against how you do the well it depends against how you do the math but like if I would say by any math but like if I would say by any math but like if I would say by any reasonable economic started if you look reasonable economic started if you look reasonable economic started if you look at the future income of humans and there at the future income of humans and there at the future income of humans and there aren't any you know that's not a good

aren't any you know that's not a good aren't any you know that's not a good investment investment investment moreover like why why can't we have a moreover like why why can't we have a moreover like why why can't we have a little bit of pride in our species damn little bit of pride in our species damn little bit of pride in our species damn it you know why should we just build it you know why should we just build it you know why should we just build another species that gets rid of us if another species that gets rid of us if another species that gets rid of us if we were Neanderthals we were Neanderthals we were Neanderthals would we really consider it a smart move would we really consider it a smart move would we really consider it a smart move if the if the if the if we had really Advanced biotech to if we had really Advanced biotech to if we had really Advanced biotech to build homo sapiens build homo sapiens build homo sapiens you you know you might say hey Max you you you know you might say hey Max you you you know you might say hey Max you know yeah let's build build the know yeah let's build build the know yeah let's build build the these Homo sapiens they're going to be these Homo sapiens they're going to be these Homo sapiens they're going to be smarter than us maybe they can help us smarter than us maybe they can help us smarter than us maybe they can help us defend us better against the Predators defend us better against the Predators defend us better against the Predators and help fix their bar caves make them and help fix their bar caves make them and help fix their bar caves make them nicer and we'll control them undoubtedly nicer and we'll control them undoubtedly nicer and we'll control them undoubtedly you know so then they build build a you know so then they build build a you know so then they build build a couple a little baby girl little baby couple a little baby girl little baby couple a little baby girl little baby boy you know and boy you know and boy you know and and then you have some some wise old and and then you have some some wise old and and then you have some some wise old and Neanderthal Elder was like hmm I'm Neanderthal Elder was like hmm I'm Neanderthal Elder was like hmm I'm scared that uh we're opening in scared that uh we're opening in scared that uh we're opening in Pandora's Box here and that we're going Pandora's Box here and that we're going Pandora's Box here and that we're going to get outsmarted by these super Neanderthal intelligences and super Neanderthal intelligences and there won't be any neanderthals left and there won't be any neanderthals left and there won't be any neanderthals left and then but then you have a bunch of others then but then you have a bunch of others then but then you have a bunch of others in the cave right yeah are you such a in the cave right yeah are you such a in the cave right yeah are you such a Luddite scaremonger of course they're Luddite scaremonger of course they're Luddite scaremonger of course they're going to want to keep us around because going to want to keep us around because going to want to keep us around because we are their creators and and why you we are their creators and and why you we are their creators and and why you know the smaller I think the smarter know the smaller I think the smarter know the smaller I think the smarter they get the nicer they're gonna get they get the nicer they're gonna get they get the nicer they're gonna get they're gonna leave us they're gonna they're gonna leave us they're gonna they're gonna leave us they're gonna they're going to want this around and they're going to want this around and they're going to want this around and it's going to be fine and and besides it's going to be fine and and besides it's going to be fine and and besides look at these babies they're so cute look at these babies they're so cute look at these babies they're so cute clearly they're totally harmless that's clearly they're totally harmless that's clearly they're totally harmless that's exact those babies are exactly gpt4 yeah exact those babies are exactly gpt4 yeah exact those babies are exactly gpt4 yeah it's not I want to be clear it's not

it's not I want to be clear it's not it's not I want to be clear it's not gpt4 gpt4 gpt4 that's terrifying it's the gpt4 is a that's terrifying it's the gpt4 is a that's terrifying it's the gpt4 is a baby technology baby technology baby technology you know and Microsoft even had a paper you know and Microsoft even had a paper you know and Microsoft even had a paper recently out recently out recently out uh with a title something like sparkles uh with a title something like sparkles uh with a title something like sparkles of AGI whatever basically saying this is of AGI whatever basically saying this is of AGI whatever basically saying this is baby AI baby AI baby AI I like these little Neanderthal babies I like these little Neanderthal babies I like these little Neanderthal babies and it's going to grow up there's going and it's going to grow up there's going and it's going to grow up there's going to be other systems from from the same to be other systems from from the same to be other systems from from the same company from other companies they'll be company from other companies they'll be company from other companies they'll be way more powerful and but they're going way more powerful and but they're going way more powerful and but they're going to take all the things to take all the things to take all the things ideas from these babies ideas from these babies ideas from these babies and before we know it we're gonna be and before we know it we're gonna be and before we know it we're gonna be like like like those last neanderthals who are pretty those last neanderthals who are pretty those last neanderthals who are pretty disappointed and when they realized that disappointed and when they realized that disappointed and when they realized that they were getting replaced well this they were getting replaced well this they were getting replaced well this interesting point you make which is the interesting point you make which is the interesting point you make which is the programming is it's entirely possible programming is it's entirely possible programming is it's entirely possible that gpt4 is already that gpt4 is already that gpt4 is already the kind of system that can the kind of system that can the kind of system that can change everything change everything change everything by writing programs sorry it's yeah it's by writing programs sorry it's yeah it's by writing programs sorry it's yeah it's because it's Life 2.0 because it's Life 2.0 because it's Life 2.0 the systems I'm afraid of are going to the systems I'm afraid of are going to the systems I'm afraid of are going to look nothing like a large language model look nothing like a large language model look nothing like a large language model and they're not and they're not and they're not but once it gets once it or other people but once it gets once it or other people but once it gets once it or other people figure out a way of using this Tech to figure out a way of using this Tech to figure out a way of using this Tech to make much better Tech right it's just make much better Tech right it's just make much better Tech right it's just constantly replacing its software and constantly replacing its software and constantly replacing its software and from everything we've seen about how how from everything we've seen about how how from everything we've seen about how how these work under the hood they're like these work under the hood they're like these work under the hood they're like the minimum viable intelligence they do the minimum viable intelligence they do the minimum viable intelligence they do everything you know the dumbest way that everything you know the dumbest way that everything you know the dumbest way that still works sort of yeah and um

still works sort of yeah and um still works sort of yeah and um so they were life 3.0 except when they so they were life 3.0 except when they so they were life 3.0 except when they replace their software it's a lot faster replace their software it's a lot faster replace their software it's a lot faster than when you when when you decide to than when you when when you decide to than when you when when you decide to learn Swedish learn Swedish learn Swedish and moreover they think a lot faster and moreover they think a lot faster and moreover they think a lot faster than us too so when uh you know we don't than us too so when uh you know we don't than us too so when uh you know we don't think uh have think uh have think uh have one one one logical step every nanosecond or a few logical step every nanosecond or a few logical step every nanosecond or a few or so the way they do and we can't also or so the way they do and we can't also or so the way they do and we can't also just suddenly scale up our Hardware just suddenly scale up our Hardware just suddenly scale up our Hardware massively in the cloud which we're so massively in the cloud which we're so massively in the cloud which we're so limited right limited right limited right so they are in it they are also life so they are in it they are also life so they are in it they are also life have have have can soon be become a little bit more can soon be become a little bit more can soon be become a little bit more like life 3.0 and that if they need more like life 3.0 and that if they need more like life 3.0 and that if they need more Hardware hey just rent it in the cloud Hardware hey just rent it in the cloud Hardware hey just rent it in the cloud you know how do you pay for it well with you know how do you pay for it well with you know how do you pay for it well with all the services you provide all the services you provide all the services you provide yeah yeah yeah and what we haven't seen yet and what we haven't seen yet and what we haven't seen yet which could change a lot is uh entire which could change a lot is uh entire which could change a lot is uh entire Software System so right now programming Software System so right now programming Software System so right now programming is done sort of in bits and pieces uh as is done sort of in bits and pieces uh as is done sort of in bits and pieces uh as as an assistant tool to humans but I do as an assistant tool to humans but I do as an assistant tool to humans but I do a lot of programming and with the kind a lot of programming and with the kind a lot of programming and with the kind of stuff that gbt4 is able to do I mean of stuff that gbt4 is able to do I mean of stuff that gbt4 is able to do I mean it's replacing a lot what I'm able to do it's replacing a lot what I'm able to do it's replacing a lot what I'm able to do but I you still need a human in the loop but I you still need a human in the loop but I you still need a human in the loop to kind of manage the design of things to kind of manage the design of things to kind of manage the design of things manage like what are the prompts that manage like what are the prompts that manage like what are the prompts that generate the kind of stuff to do some generate the kind of stuff to do some generate the kind of stuff to do some basic adjustment of the codes let's do basic adjustment of the codes let's do basic adjustment of the codes let's do some debugging but if it's possible to

some debugging but if it's possible to some debugging but if it's possible to add on top of GPT for kind of a feedback add on top of GPT for kind of a feedback add on top of GPT for kind of a feedback loop of of uh self-debugging improving loop of of uh self-debugging improving loop of of uh self-debugging improving the code and then you launch that system the code and then you launch that system the code and then you launch that system out into the wild on the internet out into the wild on the internet out into the wild on the internet because everything is connected and have because everything is connected and have because everything is connected and have it do things have it interact with it do things have it interact with it do things have it interact with humans and then get that feedback now humans and then get that feedback now humans and then get that feedback now you have this giant ecosystem yeah of you have this giant ecosystem yeah of you have this giant ecosystem yeah of humans that's one of the things that uh humans that's one of the things that uh humans that's one of the things that uh yeah Elon Musk recently sort of tweeted yeah Elon Musk recently sort of tweeted yeah Elon Musk recently sort of tweeted as a case why everyone needs to pay as a case why everyone needs to pay as a case why everyone needs to pay Seven dollars or whatever for Twitter to Seven dollars or whatever for Twitter to Seven dollars or whatever for Twitter to make sure they're real they're make sure make sure they're real they're make sure make sure they're real they're make sure they're real we're now going to be they're real we're now going to be they're real we're now going to be living in a world where the the Bots are living in a world where the the Bots are living in a world where the the Bots are getting smarter and smarter and smarter getting smarter and smarter and smarter getting smarter and smarter and smarter to a degree where you can't uh you can't to a degree where you can't uh you can't to a degree where you can't uh you can't tell the difference between a human and tell the difference between a human and tell the difference between a human and a bot that's right and now you can have a bot that's right and now you can have a bot that's right and now you can have uh Bots outnumber humans by yeah one uh Bots outnumber humans by yeah one uh Bots outnumber humans by yeah one million to one which is why he's making million to one which is why he's making million to one which is why he's making the case why you have to pay yeah to the case why you have to pay yeah to the case why you have to pay yeah to prove you're human which is one of the prove you're human which is one of the prove you're human which is one of the only mechanisms which is depressing and only mechanisms which is depressing and only mechanisms which is depressing and I yeah I feel we have to remember I yeah I feel we have to remember I yeah I feel we have to remember as individuals we should from time to as individuals we should from time to as individuals we should from time to time ask ourselves why are we doing what time ask ourselves why are we doing what time ask ourselves why are we doing what we're doing all right then as a species we're doing all right then as a species we're doing all right then as a species we need to do that too we need to do that too we need to do that too so if we're building as as you say so if we're building as as you say so if we're building as as you say machines that are outnumbering us machines that are outnumbering us machines that are outnumbering us and more and more outsmarting us and and and more and more outsmarting us and and and more and more outsmarting us and and replacing us on the job market not just replacing us on the job market not just replacing us on the job market not just for the dangerous and and boring tasks for the dangerous and and boring tasks for the dangerous and and boring tasks but also for writing poems and doing art but also for writing poems and doing art but also for writing poems and doing art and things that a lot of people find and things that a lot of people find and things that a lot of people find really meaningful God ask yourself why really meaningful God ask yourself why really meaningful God ask yourself why why are we doing this

why are we doing this why are we doing this uh we are the answer is moloch is uh we are the answer is moloch is uh we are the answer is moloch is tricking us into doing it tricking us into doing it tricking us into doing it and it's such a clever trick that even and it's such a clever trick that even and it's such a clever trick that even though we see the trick we still have no though we see the trick we still have no though we see the trick we still have no choice but to fall for it right choice but to fall for it right choice but to fall for it right come also the thing you said about you come also the thing you said about you come also the thing you said about you using uh using uh using uh co-pilot AI tools to program faster how co-pilot AI tools to program faster how co-pilot AI tools to program faster how many time what factor faster would you many time what factor faster would you many time what factor faster would you say your code now does it go twice as say your code now does it go twice as say your code now does it go twice as fast or I don't really fast or I don't really fast or I don't really uh because it's such a new tool yeah uh because it's such a new tool yeah uh because it's such a new tool yeah it's I don't know if speed is it's I don't know if speed is it's I don't know if speed is significantly improved but it feels like significantly improved but it feels like significantly improved but it feels like I'm a year away from being I'm a year away from being I'm a year away from being uh five to ten times faster so if that's uh five to ten times faster so if that's uh five to ten times faster so if that's typical for programmers then uh you're typical for programmers then uh you're typical for programmers then uh you're already seeing another kind of self already seeing another kind of self already seeing another kind of self recursive self-improvement right because recursive self-improvement right because recursive self-improvement right because previously uh one like a major previously uh one like a major previously uh one like a major generation of improvement of the codes generation of improvement of the codes generation of improvement of the codes would happen on the human r d time scale would happen on the human r d time scale would happen on the human r d time scale and now if that's five times shorter and now if that's five times shorter and now if that's five times shorter then it's going to take five times less then it's going to take five times less then it's going to take five times less time than it otherwise would to develop time than it otherwise would to develop time than it otherwise would to develop the next level of these tools and so on the next level of these tools and so on the next level of these tools and so on so this these These are the this is so this these These are the this is so this these These are the this is exactly the sort of beginning of an of exactly the sort of beginning of an of exactly the sort of beginning of an of an intelligence explosion they can be an intelligence explosion they can be an intelligence explosion they can be humans in the loop a lot in the early humans in the loop a lot in the early humans in the loop a lot in the early stages and then eventually humans are stages and then eventually humans are stages and then eventually humans are needed less and less and the machines needed less and less and the machines needed less and less and the machines can more kind of go along but you what can more kind of go along but you what can more kind of go along but you what you said there is just an exact example you said there is just an exact example you said there is just an exact example of these sort of things another thing of these sort of things another thing of these sort of things another thing which which um I was kind of lying on my psychiatrist

I was kind of lying on my psychiatrist imagining I'm on a psychiatrist's couch imagining I'm on a psychiatrist's couch imagining I'm on a psychiatrist's couch here saying what are my fears that here saying what are my fears that here saying what are my fears that people would do with um AI systems people would do with um AI systems people would do with um AI systems another another another so I mentioned three that I had fears so I mentioned three that I had fears so I mentioned three that I had fears about many years ago that they would do about many years ago that they would do about many years ago that they would do uh namely uh teacher the code uh uh namely uh teacher the code uh uh namely uh teacher the code uh connected to the internet then teach it connected to the internet then teach it connected to the internet then teach it to manipulate humans a fourth one is to manipulate humans a fourth one is to manipulate humans a fourth one is building an API building an API building an API where code can control all the super where code can control all the super where code can control all the super powerful thing right that is very powerful thing right that is very powerful thing right that is very unfortunate because one thing that unfortunate because one thing that unfortunate because one thing that systems like gpt4 have going for them is systems like gpt4 have going for them is systems like gpt4 have going for them is that they are an Oracle in the sense that they are an Oracle in the sense that they are an Oracle in the sense that they just answer questions there's that they just answer questions there's that they just answer questions there's no robot connected to the gpt4 gpt4 no robot connected to the gpt4 gpt4 no robot connected to the gpt4 gpt4 can't go and do stock trading based on can't go and do stock trading based on can't go and do stock trading based on its thinking yeah it is not an agent an its thinking yeah it is not an agent an its thinking yeah it is not an agent an intelligent agent is something that intelligent agent is something that intelligent agent is something that takes in information from the world takes in information from the world takes in information from the world processes it processes it processes it to figure out what action to take based to figure out what action to take based to figure out what action to take based on its goals that it has and then does on its goals that it has and then does on its goals that it has and then does something something something back on the world but once you have an back on the world but once you have an back on the world but once you have an API for example tpd4 Nothing Stops Joe API for example tpd4 Nothing Stops Joe API for example tpd4 Nothing Stops Joe schmoe and all and a lot of other people schmoe and all and a lot of other people schmoe and all and a lot of other people from building real agents from building real agents from building real agents which just keep making calls somewhere which just keep making calls somewhere which just keep making calls somewhere in some inner loop somewhere to these in some inner loop somewhere to these in some inner loop somewhere to these powerful Oracle systems powerful Oracle systems powerful Oracle systems and which makes them themselves much and which makes them themselves much and which makes them themselves much more powerful that's another kind of um more powerful that's another kind of um more powerful that's another kind of um unfortunate development which I think we unfortunate development which I think we unfortunate development which I think we would have been better off uh delaying I would have been better off uh delaying I would have been better off uh delaying I don't want to pick on any particular don't want to pick on any particular don't want to pick on any particular companies I think they're all under a companies I think they're all under a companies I think they're all under a lot of pressure to make money yeah

lot of pressure to make money yeah lot of pressure to make money yeah and um and um and um again we the reason we're calling for again we the reason we're calling for again we the reason we're calling for this pause is to give them all cover to this pause is to give them all cover to this pause is to give them all cover to do what they know is the right thing do what they know is the right thing do what they know is the right thing slow down a little bit at this point but slow down a little bit at this point but slow down a little bit at this point but everything we've talked about I hope everything we've talked about I hope everything we've talked about I hope will can will make it clear that people will can will make it clear that people will can will make it clear that people watching this you know why these sort of human level tools can these sort of human level tools can cause a gradual acceleration you keep cause a gradual acceleration you keep cause a gradual acceleration you keep using yesterday's technology to build using yesterday's technology to build using yesterday's technology to build tomorrow's technology yeah and when you tomorrow's technology yeah and when you tomorrow's technology yeah and when you do do do that over and over again you naturally that over and over again you naturally that over and over again you naturally get an explosion you know that's the get an explosion you know that's the get an explosion you know that's the definition of an explosion in science definition of an explosion in science definition of an explosion in science right like right like right like if you have if you have if you have two people two people two people um they fall in love now you have four um they fall in love now you have four um they fall in love now you have four people and then they can make more people and then they can make more people and then they can make more babies and now you have eight people and babies and now you have eight people and babies and now you have eight people and then then you have 16 32 64 Etc that's then then you have 16 32 64 Etc that's then then you have 16 32 64 Etc that's we call that a population explosion we call that a population explosion we call that a population explosion where it's just that each where it's just that each where it's just that each if it's instead free neutrons in a if it's instead free neutrons in a if it's instead free neutrons in a nuclear reaction that if H1 can make nuclear reaction that if H1 can make nuclear reaction that if H1 can make more than one then you get an more than one then you get an more than one then you get an exponential growth in that we call it a exponential growth in that we call it a exponential growth in that we call it a nuclear explosion all explosions are nuclear explosion all explosions are nuclear explosion all explosions are like that and an intelligence explosion like that and an intelligence explosion like that and an intelligence explosion it's just exactly the same principle it's just exactly the same principle it's just exactly the same principle that some quantity some amount of that some quantity some amount of that some quantity some amount of intelligence can make more intelligence intelligence can make more intelligence intelligence can make more intelligence than that and then repeat you always get than that and then repeat you always get than that and then repeat you always get exponentials exponentials exponentials what's your intuition why does you what's your intuition why does you what's your intuition why does you mention there's some technical reasons mention there's some technical reasons mention there's some technical reasons why it doesn't stop at a certain point why it doesn't stop at a certain point why it doesn't stop at a certain point what's your intuition and uh do you have what's your intuition and uh do you have what's your intuition and uh do you have any intuition why it might stop any intuition why it might stop any intuition why it might stop it's obviously going to stop when it

it's obviously going to stop when it it's obviously going to stop when it bumps up against the laws of physics bumps up against the laws of physics bumps up against the laws of physics there are some things you just can't do there are some things you just can't do there are some things you just can't do no matter how smart you are right no matter how smart you are right no matter how smart you are right allegedly um um laws of physics yeah yeah right Seth laws of physics yeah yeah right Seth laws of physics yeah yeah right Seth Lloyd wrote a really cool paper on the Lloyd wrote a really cool paper on the Lloyd wrote a really cool paper on the physical limits on uh computation for physical limits on uh computation for physical limits on uh computation for example if you make it example if you make it example if you make it put too much energy into it and finite put too much energy into it and finite put too much energy into it and finite space it'll turn into a black hole you space it'll turn into a black hole you space it'll turn into a black hole you can't move information around Fashion can't move information around Fashion can't move information around Fashion the speed of light stuff like that but the speed of light stuff like that but the speed of light stuff like that but uh it's hard to store uh it's hard to store uh it's hard to store way more than than a modest number bits way more than than a modest number bits way more than than a modest number bits per atom Etc but you know those limits per atom Etc but you know those limits per atom Etc but you know those limits are just astronomically above like 30 are just astronomically above like 30 are just astronomically above like 30 orders of magnitude above where we are orders of magnitude above where we are orders of magnitude above where we are now so no bigger different bigger jump in bigger different bigger jump in intelligence than if you go from uh from intelligence than if you go from uh from intelligence than if you go from uh from an ant to a human an ant to a human an ant to a human I think I think I think of course what we want to do is have of course what we want to do is have of course what we want to do is have have a controlled have a controlled have a controlled thing the nuclear reactor you put thing the nuclear reactor you put thing the nuclear reactor you put moderators in to make sure exactly it moderators in to make sure exactly it moderators in to make sure exactly it doesn't blow up out of control right doesn't blow up out of control right doesn't blow up out of control right when we do um when we do um when we do um experiments with Biology and cells and experiments with Biology and cells and experiments with Biology and cells and so on you know we also try to make sure so on you know we also try to make sure so on you know we also try to make sure it doesn't get out of control it doesn't get out of control it doesn't get out of control um um um we can do this with ai2 the thing is we we can do this with ai2 the thing is we we can do this with ai2 the thing is we haven't succeeded yet and Morlock is haven't succeeded yet and Morlock is haven't succeeded yet and Morlock is exactly exactly exactly doing the opposite just fueling just doing the opposite just fueling just doing the opposite just fueling just egging everybody on faster faster faster egging everybody on faster faster faster egging everybody on faster faster faster or the other company is going to catch or the other company is going to catch or the other company is going to catch up with you or the other country is up with you or the other country is up with you or the other country is going to catch up with you

going to catch up with you going to catch up with you really those we have to want this stuff really those we have to want this stuff really those we have to want this stuff we have and and I don't believe in this we have and and I don't believe in this we have and and I don't believe in this just asking people to look into their just asking people to look into their just asking people to look into their hearts and do the right thing it's hearts and do the right thing it's hearts and do the right thing it's easier for others to say that but like easier for others to say that but like easier for others to say that but like if if you're in a situation where your if if you're in a situation where your if if you're in a situation where your company is going to get screwed company is going to get screwed company is going to get screwed if you if you if you by other companies that are not stopping by other companies that are not stopping by other companies that are not stopping you know you're putting people in a very you know you're putting people in a very you know you're putting people in a very hard situation the the right thing to do hard situation the the right thing to do hard situation the the right thing to do is change the whole incentive structure is change the whole incentive structure is change the whole incentive structure instead instead instead and and this is not an old and and this is not an old and and this is not an old maybe I should say one more thing about maybe I should say one more thing about maybe I should say one more thing about this because molok has been around as this because molok has been around as this because molok has been around as Humanity's Humanity's Humanity's number one or number two enemy since the number one or number two enemy since the number one or number two enemy since the beginning of civilization and we came up beginning of civilization and we came up beginning of civilization and we came up with some really cool counter measures with some really cool counter measures with some really cool counter measures like first of all already over a hundred like first of all already over a hundred like first of all already over a hundred thousand years ago Evolution realized thousand years ago Evolution realized thousand years ago Evolution realized that it was very unhelpful that people that it was very unhelpful that people that it was very unhelpful that people kept killing each other all the time kept killing each other all the time kept killing each other all the time yeah so it genetically yeah so it genetically yeah so it genetically gave us compassion gave us compassion gave us compassion and made it so that it like if you get and made it so that it like if you get and made it so that it like if you get two drunk dudes getting into a pointless two drunk dudes getting into a pointless two drunk dudes getting into a pointless bar fight bar fight bar fight they might give each other black eyes they might give each other black eyes they might give each other black eyes but they have a lot of inhibition but they have a lot of inhibition but they have a lot of inhibition towards towards just killing each other towards towards just killing each other towards towards just killing each other that's a gen and similarly if you find a that's a gen and similarly if you find a that's a gen and similarly if you find a baby lying on the street when you go out baby lying on the street when you go out baby lying on the street when you go out for your morning jog tomorrow you're for your morning jog tomorrow you're for your morning jog tomorrow you're gonna stop and pick it up right even gonna stop and pick it up right even gonna stop and pick it up right even though maybe it make you late for your though maybe it make you late for your though maybe it make you late for your next podcast next podcast next podcast so Evolution gave us these genes that so Evolution gave us these genes that so Evolution gave us these genes that make our own egoistic incentives more

make our own egoistic incentives more make our own egoistic incentives more aligned with what's good for the greater aligned with what's good for the greater aligned with what's good for the greater group or part of right and then uh as we group or part of right and then uh as we group or part of right and then uh as we got a bit more sophisticated and got a bit more sophisticated and got a bit more sophisticated and developed language developed language developed language we invented gossip which is also a we invented gossip which is also a we invented gossip which is also a fantastic anti-malark right because now fantastic anti-malark right because now fantastic anti-malark right because now it it really discourages it it really discourages it it really discourages Liars Moochers cheaters because it their Liars Moochers cheaters because it their Liars Moochers cheaters because it their own incentive now is not to do this own incentive now is not to do this own incentive now is not to do this because word quickly gets around and because word quickly gets around and because word quickly gets around and then suddenly people aren't going to then suddenly people aren't going to then suddenly people aren't going to invite them to their dinners anymore or invite them to their dinners anymore or invite them to their dinners anymore or trust them and then when we got still trust them and then when we got still trust them and then when we got still more sophisticated and bigger societies more sophisticated and bigger societies more sophisticated and bigger societies you know invented the legal system you know invented the legal system you know invented the legal system where even strangers who didn't couldn't where even strangers who didn't couldn't where even strangers who didn't couldn't rely on gossip and and things like this rely on gossip and and things like this rely on gossip and and things like this would treat each other would have an would treat each other would have an would treat each other would have an incentive now those guys in the bar incentive now those guys in the bar incentive now those guys in the bar fights even if they someone is so drunk fights even if they someone is so drunk fights even if they someone is so drunk that he that he that he actually wants to kill the other guy actually wants to kill the other guy actually wants to kill the other guy he also has a little thought on the back he also has a little thought on the back he also has a little thought on the back of his head that you know do I really of his head that you know do I really of his head that you know do I really want to spend the next 10 years want to spend the next 10 years want to spend the next 10 years eating like really crappy food in a eating like really crappy food in a eating like really crappy food in a small room uh I'm just gonna I'm just small room uh I'm just gonna I'm just small room uh I'm just gonna I'm just gonna chill out you know so and we we gonna chill out you know so and we we gonna chill out you know so and we we similarly have tried to give these similarly have tried to give these similarly have tried to give these incentives to our corporations by having incentives to our corporations by having incentives to our corporations by having having regulation and all sorts of having regulation and all sorts of having regulation and all sorts of oversight so that their incentives are oversight so that their incentives are oversight so that their incentives are aligned with the greater good we've aligned with the greater good we've aligned with the greater good we've tried really hard tried really hard tried really hard um and um the the big problem that we're um and um the the big problem that we're um and um the the big problem that we're failing now failing now failing now is not that we haven't tried before but is not that we haven't tried before but is not that we haven't tried before but it's just that the tech is growing much it's just that the tech is growing much it's just that the tech is growing much is developing much faster than The

is developing much faster than The is developing much faster than The Regulators been able to keep up right so Regulators been able to keep up right so Regulators been able to keep up right so Regulators it's kind of comical the Regulators it's kind of comical the Regulators it's kind of comical the European Union right now is doing this European Union right now is doing this European Union right now is doing this AI Act Right which and in the beginning AI Act Right which and in the beginning AI Act Right which and in the beginning they had a little opt-out exception that they had a little opt-out exception that they had a little opt-out exception that gpt4 would be completely excluded from gpt4 would be completely excluded from gpt4 would be completely excluded from regulation regulation regulation brilliant idea what's the logic behind brilliant idea what's the logic behind brilliant idea what's the logic behind that that that some lobbyists post successfully for some lobbyists post successfully for some lobbyists post successfully for this so we were actually quite involved this so we were actually quite involved this so we were actually quite involved with the future Life Institute with the future Life Institute with the future Life Institute um Mark Bracco Mr ook Anthony gear and um Mark Bracco Mr ook Anthony gear and um Mark Bracco Mr ook Anthony gear and others you know we're quite involved others you know we're quite involved others you know we're quite involved with um talking to very educating with um talking to very educating with um talking to very educating various people involved in this process various people involved in this process various people involved in this process about about about these general purpose AI models coming these general purpose AI models coming these general purpose AI models coming and pointing out that they would become and pointing out that they would become and pointing out that they would become the laughing stock if they didn't put it the laughing stock if they didn't put it the laughing stock if they didn't put it in so it the French started pushing for in so it the French started pushing for in so it the French started pushing for it it got put in to the draft and it it it got put in to the draft and it it it got put in to the draft and it looked like all was good and then there looked like all was good and then there looked like all was good and then there was a huge counter push from lobbyists was a huge counter push from lobbyists was a huge counter push from lobbyists yeah there were more lobbyists in yeah there were more lobbyists in yeah there were more lobbyists in Brussels from tech companies and from Brussels from tech companies and from Brussels from tech companies and from oil companies for example and it looked oil companies for example and it looked oil companies for example and it looked like it might it's going to maybe get like it might it's going to maybe get like it might it's going to maybe get taken out again taken out again taken out again and now gpt4 happened and now gpt4 happened and now gpt4 happened and I think it's going to stay in but and I think it's going to stay in but and I think it's going to stay in but this just shows you know monologue can this just shows you know monologue can this just shows you know monologue can be defeated be defeated be defeated but the the challenge we're facing is but the the challenge we're facing is but the the challenge we're facing is that the tech is generally much faster that the tech is generally much faster that the tech is generally much faster than what the than what the than what the policymakers are policymakers are policymakers are and a lot of the policy makers also and a lot of the policy makers also and a lot of the policy makers also don't have a tech background so it's you don't have a tech background so it's you don't have a tech background so it's you know we really need to work hard to know we really need to work hard to know we really need to work hard to educate them on on how on what's taking

educate them on on how on what's taking educate them on on how on what's taking place here place here place here so so we're getting the situation where so so we're getting the situation where so so we're getting the situation where the first kind of non so you know I the first kind of non so you know I the first kind of non so you know I Define artificial intelligence just as Define artificial intelligence just as Define artificial intelligence just as non-biological intelligence right non-biological intelligence right non-biological intelligence right and by that definition and by that definition and by that definition a company a corporation is also an a company a corporation is also an a company a corporation is also an artificial intelligence because the artificial intelligence because the artificial intelligence because the corporation isn't it's humans it's the corporation isn't it's humans it's the corporation isn't it's humans it's the system system system if its CEO decides the CEO of a tobacco if its CEO decides the CEO of a tobacco if its CEO decides the CEO of a tobacco company decides one morning that she or company decides one morning that she or company decides one morning that she or he doesn't want to sell cigarettes he doesn't want to sell cigarettes he doesn't want to sell cigarettes anymore they'll just put another CEO in anymore they'll just put another CEO in anymore they'll just put another CEO in there there there it's not enough to align the incentives it's not enough to align the incentives it's not enough to align the incentives of individual people or in align of individual people or in align of individual people or in align individual computers incentives individual computers incentives individual computers incentives to their owners which is what to their owners which is what to their owners which is what technically AI Safety Research is about technically AI Safety Research is about technically AI Safety Research is about you also have to align the incentives of you also have to align the incentives of you also have to align the incentives of Corporations Corporations Corporations with a greater good and some with a greater good and some with a greater good and some corporations have gotten so big and so corporations have gotten so big and so corporations have gotten so big and so powerful very quickly and that in many powerful very quickly and that in many powerful very quickly and that in many cases they're lobbyists instead align cases they're lobbyists instead align cases they're lobbyists instead align The Regulators to what they want rather The Regulators to what they want rather The Regulators to what they want rather than the other way around the classic than the other way around the classic than the other way around the classic regulatory capture right is is uh the regulatory capture right is is uh the regulatory capture right is is uh the thing that the Slowdown hopes to achieve thing that the Slowdown hopes to achieve thing that the Slowdown hopes to achieve is give enough time to the regulars to is give enough time to the regulars to is give enough time to the regulars to catch up or enough time to the companies catch up or enough time to the companies catch up or enough time to the companies themselves to breathe and understand how themselves to breathe and understand how themselves to breathe and understand how to do AI safety correctly I think both to do AI safety correctly I think both to do AI safety correctly I think both and but I think that the vision the path and but I think that the vision the path and but I think that the vision the path to success I see is first you give a to success I see is first you give a to success I see is first you give a breather actually to to the people in breather actually to to the people in breather actually to to the people in these companies their leadership who these companies their leadership who these companies their leadership who wants to do the right thing and they all wants to do the right thing and they all wants to do the right thing and they all have safety teams and so on other have safety teams and so on other have safety teams and so on other companies give them a chance to

companies give them a chance to companies give them a chance to get together with the other companies get together with the other companies get together with the other companies and the outside pressure can also help and the outside pressure can also help and the outside pressure can also help catalyze that right catalyze that right catalyze that right and and work out and and work out and and work out what is it that's what is it that's what is it that's what are the reasonable safety what are the reasonable safety what are the reasonable safety requirements one should put on future requirements one should put on future requirements one should put on future systems before they get rolled out systems before they get rolled out systems before they get rolled out there are a lot of people also in there are a lot of people also in there are a lot of people also in Academia and elsewhere outside of these Academia and elsewhere outside of these Academia and elsewhere outside of these companies who can be brought into this companies who can be brought into this companies who can be brought into this and have a lot of very good ideas and have a lot of very good ideas and have a lot of very good ideas and then um and then um and then um I think it's very realistic that within I think it's very realistic that within I think it's very realistic that within six months six months six months you can get these people coming up so you can get these people coming up so you can get these people coming up so here's a white paper here's where we all here's a white paper here's where we all here's a white paper here's where we all think it's reasonable think it's reasonable think it's reasonable um you know you didn't just because cars um you know you didn't just because cars um you know you didn't just because cars killed a lot of people they didn't ban killed a lot of people they didn't ban killed a lot of people they didn't ban cars but they got together a bunch of cars but they got together a bunch of cars but they got together a bunch of people and decided you know in order to people and decided you know in order to people and decided you know in order to be allowed to sell a car it has to have be allowed to sell a car it has to have be allowed to sell a car it has to have a seat belt in them a seat belt in them a seat belt in them they're the analogous things that you they're the analogous things that you they're the analogous things that you can start requiring a future AI systems can start requiring a future AI systems can start requiring a future AI systems so that they are are safe so that they are are safe so that they are are safe and uh once this have this heavy lifting and uh once this have this heavy lifting and uh once this have this heavy lifting this intellectual work has been done by this intellectual work has been done by this intellectual work has been done by experts in the field which can be done experts in the field which can be done experts in the field which can be done quickly quickly quickly I think it's beginning to be quite easy I think it's beginning to be quite easy I think it's beginning to be quite easy to get policy makers to to get policy makers to to get policy makers to to see yeah this is a good idea and it's to see yeah this is a good idea and it's to see yeah this is a good idea and it's it's you know it's you know it's you know for the fight for the companies to fight for the fight for the companies to fight for the fight for the companies to fight molok molok molok they want and I I believe Sam Altman has they want and I I believe Sam Altman has they want and I I believe Sam Altman has explicitly called for this they want The explicitly called for this they want The explicitly called for this they want The Regulators to actually adopt it so that Regulators to actually adopt it so that Regulators to actually adopt it so that their competition is going to abide by their competition is going to abide by their competition is going to abide by it too right you don't want

it too right you don't want it too right you don't want uh uh uh you don't want to be invade enacting all you don't want to be invade enacting all you don't want to be invade enacting all these principles then you abide by them these principles then you abide by them these principles then you abide by them and then there's this one little and then there's this one little and then there's this one little company that company that company that doesn't sign on to it and then now they doesn't sign on to it and then now they doesn't sign on to it and then now they can gradually overtake you then the can gradually overtake you then the can gradually overtake you then the companies will get companies will get companies will get be able to sleep it's secure knowing be able to sleep it's secure knowing be able to sleep it's secure knowing that everybody's playing by the same that everybody's playing by the same that everybody's playing by the same rules so do you think it's possible to rules so do you think it's possible to rules so do you think it's possible to develop guard rails develop guard rails develop guard rails that keep the systems from that keep the systems from that keep the systems from uh for basically damaging irreparably uh for basically damaging irreparably uh for basically damaging irreparably Humanity while still enabling sort of Humanity while still enabling sort of Humanity while still enabling sort of the capitalist fueled competition the capitalist fueled competition the capitalist fueled competition between companies as they develop how to between companies as they develop how to between companies as they develop how to best make money with this AI you think best make money with this AI you think best make money with this AI you think there's a balancing totally that's there's a balancing totally that's there's a balancing totally that's possible absolutely I mean we've seen possible absolutely I mean we've seen possible absolutely I mean we've seen that in many other sectors where you've that in many other sectors where you've that in many other sectors where you've had the free market produce quite good had the free market produce quite good had the free market produce quite good things without uh causing particular things without uh causing particular things without uh causing particular harm harm harm um when the guardrails are there and um when the guardrails are there and um when the guardrails are there and they work they work they work you know you know you know capitalism is very effective good way of capitalism is very effective good way of capitalism is very effective good way of optimizing for just getting the same optimizing for just getting the same optimizing for just getting the same things done more efficiently it was but things done more efficiently it was but things done more efficiently it was but it was good you know and like in it was good you know and like in it was good you know and like in hindsight and I never met anyone hindsight and I never met anyone hindsight and I never met anyone even on parties way over on the right in in on parties way over on the right in in any country who think it was a bad any country who think it was a bad any country who think it was a bad thinks it was a terrible idea to ban thinks it was a terrible idea to ban thinks it was a terrible idea to ban child labor for example so child labor for example so child labor for example so yeah but it seems like this particular yeah but it seems like this particular yeah but it seems like this particular technology has gotten so good so fast technology has gotten so good so fast technology has gotten so good so fast become become become powerful powerful powerful to a degree where you could see in the to a degree where you could see in the to a degree where you could see in the near term the ability to make a lot of

near term the ability to make a lot of near term the ability to make a lot of money and to put guard rails develop money and to put guard rails develop money and to put guard rails develop guard rails quickly in that kind of guard rails quickly in that kind of guard rails quickly in that kind of contact seems to be tricky it's not uh contact seems to be tricky it's not uh contact seems to be tricky it's not uh similar to cars or child labor it seems similar to cars or child labor it seems similar to cars or child labor it seems like the opportunity to make a lot of like the opportunity to make a lot of like the opportunity to make a lot of money here very quickly is right here money here very quickly is right here money here very quickly is right here before so again there's this cliff on the ground you can pick up or on the ground you can pick up or whatever so you want to drive there very whatever so you want to drive there very whatever so you want to drive there very fast but it's not in anyone's incentive fast but it's not in anyone's incentive fast but it's not in anyone's incentive that we go over the cliff and it's not that we go over the cliff and it's not that we go over the cliff and it's not like everybody's in their own car all like everybody's in their own car all like everybody's in their own car all the cars are connected together with a the cars are connected together with a the cars are connected together with a chain yeah so if anyone goes over chain yeah so if anyone goes over chain yeah so if anyone goes over they'll start dragging others down the they'll start dragging others down the they'll start dragging others down the other's down too and so ultimately it's other's down too and so ultimately it's other's down too and so ultimately it's in the selfish in the selfish in the selfish interests also of the people in the interests also of the people in the interests also of the people in the companies to to companies to to companies to to slow down when the when it just start slow down when the when it just start slow down when the when it just start seeing the characters of The Cliff there seeing the characters of The Cliff there seeing the characters of The Cliff there in front of you right and the problem is in front of you right and the problem is in front of you right and the problem is that um even though the people who are that um even though the people who are that um even though the people who are building the technology building the technology building the technology and the CEOs they really get it the and the CEOs they really get it the and the CEOs they really get it the shareholders and these other Market shareholders and these other Market shareholders and these other Market forces they are people who don't forces they are people who don't forces they are people who don't honestly honestly honestly understand that the cliff is there they understand that the cliff is there they understand that the cliff is there they usually don't you have to get quite into usually don't you have to get quite into usually don't you have to get quite into the weeds to really appreciate how the weeds to really appreciate how the weeds to really appreciate how powerful this is and how fast and a lot powerful this is and how fast and a lot powerful this is and how fast and a lot of people are even still stuck again in of people are even still stuck again in of people are even still stuck again in this idea that this idea that this idea that intelligence in this carbon chauvinism intelligence in this carbon chauvinism intelligence in this carbon chauvinism as I like to call that that you can only as I like to call that that you can only as I like to call that that you can only have our level of intelligence in have our level of intelligence in have our level of intelligence in humans that there's something magical humans that there's something magical humans that there's something magical about it whereas the people in the tech

about it whereas the people in the tech about it whereas the people in the tech companies companies companies who build this stuff they all realize who build this stuff they all realize who build this stuff they all realize that intelligence is that intelligence is that intelligence is information processing of a certain kind information processing of a certain kind information processing of a certain kind and it really doesn't matter at all and it really doesn't matter at all and it really doesn't matter at all whether the information is processed by whether the information is processed by whether the information is processed by carbon atoms in neurons in brains or by carbon atoms in neurons in brains or by carbon atoms in neurons in brains or by silicon atoms and some technology we silicon atoms and some technology we silicon atoms and some technology we build build build you brought up capitalism earlier and you brought up capitalism earlier and you brought up capitalism earlier and there are a lot of people who love there are a lot of people who love there are a lot of people who love capitalism and a lot of people capitalism and a lot of people capitalism and a lot of people who really really who really really who really really don't and don't and don't and it struck me recently that the what's it struck me recently that the what's it struck me recently that the what's happening with capitalism here is happening with capitalism here is happening with capitalism here is exactly analogous to the way in which exactly analogous to the way in which exactly analogous to the way in which Super intelligence might wipe us out Super intelligence might wipe us out Super intelligence might wipe us out so so so you know you know I studied economics you know you know I studied economics you know you know I studied economics for my undergrad Stockholm School of for my undergrad Stockholm School of for my undergrad Stockholm School of Economics yay Economics yay Economics yay well no no I tell me someone's very well no no I tell me someone's very well no no I tell me someone's very interested in how how you could use interested in how how you could use interested in how how you could use Market forces to just get stuff done Market forces to just get stuff done Market forces to just get stuff done more efficiently but give the right more efficiently but give the right more efficiently but give the right incentives to Market so that it wouldn't incentives to Market so that it wouldn't incentives to Market so that it wouldn't do really bad things do really bad things do really bad things so Dylan had filmonell who's a a so Dylan had filmonell who's a a so Dylan had filmonell who's a a professor and colleague of mine at MIT professor and colleague of mine at MIT professor and colleague of mine at MIT wrote this really interesting paper with wrote this really interesting paper with wrote this really interesting paper with some collaborators recently some collaborators recently some collaborators recently where they proved mathematically that if where they proved mathematically that if where they proved mathematically that if you just up take one goal that you just you just up take one goal that you just you just up take one goal that you just optimized for optimized for optimized for on and on and on indefinitely on and on and on indefinitely on and on and on indefinitely do you think it's gonna do you think it's gonna do you think it's gonna bring you in the right direction bring you in the right direction bring you in the right direction what basically always happens is in the what basically always happens is in the what basically always happens is in the beginning it will make things better for

beginning it will make things better for beginning it will make things better for you you you but if you keep going at some point but if you keep going at some point but if you keep going at some point that's going to start making things that's going to start making things that's going to start making things worse for you again and then gradually worse for you again and then gradually worse for you again and then gradually it's going to make it really really it's going to make it really really it's going to make it really really terrible so just as a simple terrible so just as a simple terrible so just as a simple the way I think of the proof is like the way I think of the proof is like the way I think of the proof is like suppose you want to go from here suppose you want to go from here suppose you want to go from here back to Austin for example and you're back to Austin for example and you're back to Austin for example and you're like okay yeah let's just let's go south like okay yeah let's just let's go south like okay yeah let's just let's go south but you put in exactly the right sort of but you put in exactly the right sort of but you put in exactly the right sort of the right direction just optimize that the right direction just optimize that the right direction just optimize that South as possible you get closer and South as possible you get closer and South as possible you get closer and closer to Austin closer to Austin closer to Austin but uh you there's always some little but uh you there's always some little but uh you there's always some little error So you you're not going exactly error So you you're not going exactly error So you you're not going exactly towards Austin but you get pretty close towards Austin but you get pretty close towards Austin but you get pretty close but eventually you start going away but eventually you start going away but eventually you start going away again and eventually you're going to be again and eventually you're going to be again and eventually you're going to be leaving the solar system yeah and they leaving the solar system yeah and they leaving the solar system yeah and they they proved it's a beautiful they proved it's a beautiful they proved it's a beautiful mathematical proof this happens mathematical proof this happens mathematical proof this happens generally generally generally and this is very important for AI and this is very important for AI and this is very important for AI because for even though Stuart Russell because for even though Stuart Russell because for even though Stuart Russell has has has written a book and given a lot of talks written a book and given a lot of talks written a book and given a lot of talks on why it's a bad idea to have ai just on why it's a bad idea to have ai just on why it's a bad idea to have ai just blindly optimize something that's what blindly optimize something that's what blindly optimize something that's what pretty much all our systems do yeah we pretty much all our systems do yeah we pretty much all our systems do yeah we have something called the loss function have something called the loss function have something called the loss function that we're just minimizing or reward that we're just minimizing or reward that we're just minimizing or reward function we just minimize maximizing function we just minimize maximizing function we just minimize maximizing stuff stuff stuff and um capitalism is exactly like that too we capitalism is exactly like that too we want we wanted to get stuff done more want we wanted to get stuff done more want we wanted to get stuff done more efficiently that people wanted so efficiently that people wanted so efficiently that people wanted so introduce the free market things got done much more efficiently things got done much more efficiently than they did in than they did in than they did in say communism right and it got better

say communism right and it got better say communism right and it got better but then it just kept optimizing it and but then it just kept optimizing it and but then it just kept optimizing it and kept optimizing and you got every bigger kept optimizing and you got every bigger kept optimizing and you got every bigger companies and every more efficient companies and every more efficient companies and every more efficient information processing and now also very information processing and now also very information processing and now also very much powered by it much powered by it much powered by it and uh eventually a lot of people are and uh eventually a lot of people are and uh eventually a lot of people are beginning to feel wait we're kind of beginning to feel wait we're kind of beginning to feel wait we're kind of optimizing a bit too much like why did optimizing a bit too much like why did optimizing a bit too much like why did we just chop down half the rainforest we just chop down half the rainforest we just chop down half the rainforest you know and why why did suddenly these you know and why why did suddenly these you know and why why did suddenly these Regulators get Regulators get Regulators get captured by lobbyists and so on it's captured by lobbyists and so on it's captured by lobbyists and so on it's just the same optimization that's been just the same optimization that's been just the same optimization that's been running for too long running for too long running for too long if you have an AI that actually has if you have an AI that actually has if you have an AI that actually has power over the world and you just give power over the world and you just give power over the world and you just give it one goal and just like keep it one goal and just like keep it one goal and just like keep optimizing that most likely everybody's optimizing that most likely everybody's optimizing that most likely everybody's gonna be like Yay this is great in the gonna be like Yay this is great in the gonna be like Yay this is great in the beginning things are getting better beginning things are getting better beginning things are getting better but um but um but um it's almost impossible to give it it's almost impossible to give it it's almost impossible to give it exactly the right direction to optimize exactly the right direction to optimize exactly the right direction to optimize in and then in and then in and then eventually all hey Breaks Loose right eventually all hey Breaks Loose right eventually all hey Breaks Loose right Nick Bostrom and others are given the Nick Bostrom and others are given the Nick Bostrom and others are given the example to sound quite silly like what example to sound quite silly like what example to sound quite silly like what if you just want to like tell it if you just want to like tell it if you just want to like tell it cure cancer or something and that's all cure cancer or something and that's all cure cancer or something and that's all you tell it maybe it's going to decide you tell it maybe it's going to decide you tell it maybe it's going to decide to to to take over entire continents just so we take over entire continents just so we take over entire continents just so we can get more super computer facilities can get more super computer facilities can get more super computer facilities in there and in there and in there and figure out how to cure cancer backwards figure out how to cure cancer backwards figure out how to cure cancer backwards and then you're like wait that's not and then you're like wait that's not and then you're like wait that's not what I want then right and what I want then right and what I want then right and um um um the the the issue with capitalism and the the the issue with capitalism and the the the issue with capitalism and the issue with runaway I have kind of the issue with runaway I have kind of the issue with runaway I have kind of merged now because the malloc I talked merged now because the malloc I talked merged now because the malloc I talked about about about is exactly the capitalist molloch that is exactly the capitalist molloch that is exactly the capitalist molloch that we have built an economy that has

we have built an economy that has we have built an economy that has optimizing for only one thing optimizing for only one thing optimizing for only one thing profit profit profit right and that worked great back when right and that worked great back when right and that worked great back when things were very inefficient and then things were very inefficient and then things were very inefficient and then now it's getting done better and it now it's getting done better and it now it's getting done better and it worked great as long as the companies worked great as long as the companies worked great as long as the companies were small enough that they couldn't were small enough that they couldn't were small enough that they couldn't capture the regulators capture the regulators capture the regulators but but but that's not true anymore but they keep that's not true anymore but they keep that's not true anymore but they keep optimizing optimizing optimizing and now and now and now they realize that that they can these they realize that that they can these they realize that that they can these companies can make even more profit by companies can make even more profit by companies can make even more profit by building ever more powerful AI even if building ever more powerful AI even if building ever more powerful AI even if it's Reckless but optimize more and more and more and but optimize more and more and more and more and more more and more more and more so this is molok again showing up and I so this is molok again showing up and I so this is molok again showing up and I just want to anyone here who has any just want to anyone here who has any just want to anyone here who has any concerns about about uh late stage concerns about about uh late stage concerns about about uh late stage capitalism having gone a little too far capitalism having gone a little too far capitalism having gone a little too far you should worry about you should worry about you should worry about super intelligence because it's the same super intelligence because it's the same super intelligence because it's the same villain in both cases it's more like and villain in both cases it's more like and villain in both cases it's more like and optimizing one objective function optimizing one objective function optimizing one objective function aggressively blindly is going to take us aggressively blindly is going to take us aggressively blindly is going to take us there yeah we have this pause from time there yeah we have this pause from time there yeah we have this pause from time to time and look into our hearts and ask to time and look into our hearts and ask to time and look into our hearts and ask why are we doing this is this uh am I why are we doing this is this uh am I why are we doing this is this uh am I still going towards Austin or have I still going towards Austin or have I still going towards Austin or have I gone too far you know maybe we should gone too far you know maybe we should gone too far you know maybe we should change direction change direction change direction and that is the idea behind the halt for and that is the idea behind the halt for and that is the idea behind the halt for six months what six months it seems like six months what six months it seems like six months what six months it seems like a very short period just can we just a very short period just can we just a very short period just can we just linger and explore different ideas here linger and explore different ideas here linger and explore different ideas here because this feels like a really because this feels like a really because this feels like a really important moment in human history where important moment in human history where important moment in human history where pausing would actually have a

pausing would actually have a pausing would actually have a significant positive effect significant positive effect significant positive effect we said six months because we figured we said six months because we figured we said six months because we figured the number one pushback that we're gonna the number one pushback that we're gonna the number one pushback that we're gonna get in the west was like but China get in the west was like but China get in the west was like but China and and and everybody knows there's no way that everybody knows there's no way that everybody knows there's no way that China is going to catch up with the West China is going to catch up with the West China is going to catch up with the West on this in six months so it's that on this in six months so it's that on this in six months so it's that argument goes off the table and you can argument goes off the table and you can argument goes off the table and you can forget about geopolitical competition forget about geopolitical competition forget about geopolitical competition and just focus on and just focus on and just focus on the real issue that's why we put this the real issue that's why we put this the real issue that's why we put this that's really interesting but you've that's really interesting but you've that's really interesting but you've already made the case that uh even for already made the case that uh even for already made the case that uh even for China if you actually want to take on China if you actually want to take on China if you actually want to take on that argument China too would not be that argument China too would not be that argument China too would not be bothered by a longer halt because they bothered by a longer halt because they bothered by a longer halt because they don't want to lose control even more don't want to lose control even more don't want to lose control even more than the West doesn't than the West doesn't than the West doesn't that's what I think that's a really that's what I think that's a really that's what I think that's a really interesting argument like I have to interesting argument like I have to interesting argument like I have to actually really think about that which actually really think about that which actually really think about that which the the kind of thing people assume is the the kind of thing people assume is the the kind of thing people assume is if you develop an AGI if you develop an AGI if you develop an AGI that open AI if they're the ones that do that open AI if they're the ones that do that open AI if they're the ones that do it for example they're going to win but it for example they're going to win but it for example they're going to win but you're saying no they're everybody loses you're saying no they're everybody loses you're saying no they're everybody loses yeah it's gonna get better and better yeah it's gonna get better and better yeah it's gonna get better and better and better and then Kaboom we all lose and better and then Kaboom we all lose and better and then Kaboom we all lose that's what's gonna happen when lose and that's what's gonna happen when lose and that's what's gonna happen when lose and win are defined in a metric of basically win are defined in a metric of basically win are defined in a metric of basically quality of life quality of life quality of life for human civilization and for Sam for human civilization and for Sam for human civilization and for Sam Altman Altman Altman to be blunt my personal guess you know to be blunt my personal guess you know to be blunt my personal guess you know and people can quibble with this is that and people can quibble with this is that and people can quibble with this is that we're just gonna they won't be any we're just gonna they won't be any we're just gonna they won't be any humans that's it that's what I mean by humans that's it that's what I mean by humans that's it that's what I mean by lose lose lose you know if you if we can see in history you know if you if we can see in history you know if you if we can see in history once you have some species or some group

once you have some species or some group once you have some species or some group of people who aren't needed anymore of people who aren't needed anymore of people who aren't needed anymore it doesn't usually work out so well for it doesn't usually work out so well for it doesn't usually work out so well for them right yeah them right yeah them right yeah there were a lot of horses for that were there were a lot of horses for that were there were a lot of horses for that were used for traffic in Boston and then the used for traffic in Boston and then the used for traffic in Boston and then the car got invented and most of them got car got invented and most of them got car got invented and most of them got you know well we don't need to go there you know well we don't need to go there you know well we don't need to go there and uh if you look at them humans you know right now we why did the humans you know right now we why did the labor movement succeed and after the labor movement succeed and after the labor movement succeed and after the Industrial Revolution because it was Industrial Revolution because it was Industrial Revolution because it was needed needed needed even though we had a lot of mallocs and even though we had a lot of mallocs and even though we had a lot of mallocs and there was child labor and so on you know there was child labor and so on you know there was child labor and so on you know the company still needed to have workers the company still needed to have workers the company still needed to have workers and that's why strikes had power and so and that's why strikes had power and so and that's why strikes had power and so on if we get to the point where most on if we get to the point where most on if we get to the point where most humans aren't needed anymore I think humans aren't needed anymore I think humans aren't needed anymore I think it's like it's quite naive to think that it's like it's quite naive to think that it's like it's quite naive to think that they're going to still be treated well they're going to still be treated well they're going to still be treated well you know we say that yeah yeah um you know we say that yeah yeah um you know we say that yeah yeah um everybody's equal and the government everybody's equal and the government everybody's equal and the government will always will always protect them but will always will always protect them but will always will always protect them but if you look in practice if you look in practice if you look in practice groups that are very disenfranchised and groups that are very disenfranchised and groups that are very disenfranchised and don't have any actual power it usually gets screwed it usually gets screwed and uh now in in the beginning so and uh now in in the beginning so and uh now in in the beginning so Industrial Revolution Industrial Revolution Industrial Revolution we automated away muscle work we automated away muscle work we automated away muscle work but that got went worked out pretty well but that got went worked out pretty well but that got went worked out pretty well eventually because we educated ourselves eventually because we educated ourselves eventually because we educated ourselves and started doing working with our and started doing working with our and started doing working with our brains instead and got usually more brains instead and got usually more brains instead and got usually more interesting better paid jobs interesting better paid jobs interesting better paid jobs but now we're beginning to replace brain but now we're beginning to replace brain but now we're beginning to replace brain work so we've replaced a lot of boring

work so we've replaced a lot of boring work so we've replaced a lot of boring stuff like we got the pocket calculator stuff like we got the pocket calculator stuff like we got the pocket calculator so you don't have people adding so you don't have people adding so you don't have people adding multiplying numbers anymore at work fine multiplying numbers anymore at work fine multiplying numbers anymore at work fine they were better jobs they could get but they were better jobs they could get but they were better jobs they could get but now dpt4 you know now dpt4 you know now dpt4 you know and the stable diffusion and techniques and the stable diffusion and techniques and the stable diffusion and techniques like this they're really beginning to like this they're really beginning to like this they're really beginning to blow away some real some jobs that blow away some real some jobs that blow away some real some jobs that people really loved having it was a people really loved having it was a people really loved having it was a heartbreaking article just post just heartbreaking article just post just heartbreaking article just post just yesterday on social media I saw about yesterday on social media I saw about yesterday on social media I saw about this guy who was doing 3D modeling for this guy who was doing 3D modeling for this guy who was doing 3D modeling for gaming and if gaming and if gaming and if and all of a sudden now they got this and all of a sudden now they got this and all of a sudden now they got this new software you just give says prompts new software you just give says prompts new software you just give says prompts and he feels his whole job that he loved and he feels his whole job that he loved and he feels his whole job that he loved it's lost its meaning you know and uh it's lost its meaning you know and uh it's lost its meaning you know and uh I asked the gpt4 to rewrite Twinkle I asked the gpt4 to rewrite Twinkle I asked the gpt4 to rewrite Twinkle Twinkle Little Star in the style of Twinkle Little Star in the style of Twinkle Little Star in the style of Shakespeare Shakespeare Shakespeare I couldn't have done such a good job it I couldn't have done such a good job it I couldn't have done such a good job it was just really impressive you've seen a was just really impressive you've seen a was just really impressive you've seen a lot of the Arts coming out here right so lot of the Arts coming out here right so lot of the Arts coming out here right so I'm all for I'm all for I'm all for automating away the dangerous jobs and automating away the dangerous jobs and automating away the dangerous jobs and the boring jobs but I think um the boring jobs but I think um the boring jobs but I think um you hear some arguments which are too you hear some arguments which are too you hear some arguments which are too glib sometimes people say well that's glib sometimes people say well that's glib sometimes people say well that's all that's going to happen we're getting all that's going to happen we're getting all that's going to happen we're getting rid of the boring boring uh tedious rid of the boring boring uh tedious rid of the boring boring uh tedious dangerous jobs it's just not true there dangerous jobs it's just not true there dangerous jobs it's just not true there are a lot of really interesting jobs are a lot of really interesting jobs are a lot of really interesting jobs that are being taken away now journalism that are being taken away now journalism that are being taken away now journalism is getting going to get crushed uh is getting going to get crushed uh is getting going to get crushed uh coding is gonna get crushed like I coding is gonna get crushed like I coding is gonna get crushed like I predict uh the job market for predict uh the job market for predict uh the job market for programmers salaries are going to start programmers salaries are going to start programmers salaries are going to start dropping dropping dropping you know if you said you can code five you know if you said you can code five you know if you said you can code five times faster you know then you need five

times faster you know then you need five times faster you know then you need five times fewer programmers maybe there will times fewer programmers maybe there will times fewer programmers maybe there will be be be more output also but you'll still end up more output also but you'll still end up more output also but you'll still end up using fewer programs needing fewer using fewer programs needing fewer using fewer programs needing fewer programmers than today and I love coding programmers than today and I love coding programmers than today and I love coding you know I I think it's super cool you know I I think it's super cool you know I I think it's super cool um so we we need to stop and ask um so we we need to stop and ask um so we we need to stop and ask ourselves why again are we doing this as ourselves why again are we doing this as ourselves why again are we doing this as humans like humans like humans like I feel that AI should be built by I feel that AI should be built by I feel that AI should be built by Humanity for Humanity and let's not Humanity for Humanity and let's not Humanity for Humanity and let's not forget that it shouldn't be by malloc forget that it shouldn't be by malloc forget that it shouldn't be by malloc for molok or what it really is now is for molok or what it really is now is for molok or what it really is now is kind of by Humanity for moloch which kind of by Humanity for moloch which kind of by Humanity for moloch which doesn't make any sense it's for us that doesn't make any sense it's for us that doesn't make any sense it's for us that we're doing it then and um we're doing it then and um we're doing it then and um it would make a lot more sense if we it would make a lot more sense if we it would make a lot more sense if we build the develop figure out gradually build the develop figure out gradually build the develop figure out gradually safely how to make all this Tech and safely how to make all this Tech and safely how to make all this Tech and then we think about what are the kind of then we think about what are the kind of then we think about what are the kind of jobs that people really don't want to jobs that people really don't want to jobs that people really don't want to have you know and automate them all the have you know and automate them all the have you know and automate them all the way and then we ask what are the jobs way and then we ask what are the jobs way and then we ask what are the jobs that that that people really find meaning in like maybe taking care of children in the daycare taking care of children in the daycare center maybe doing art etc etc and and center maybe doing art etc etc and and center maybe doing art etc etc and and even if it were possible to automate even if it were possible to automate even if it were possible to automate that away but we don't need to do that that away but we don't need to do that that away but we don't need to do that right we built these machines right we built these machines right we built these machines well it's possible that we well it's possible that we well it's possible that we redefine or ReDiscover what are the jobs redefine or ReDiscover what are the jobs redefine or ReDiscover what are the jobs that give us meaning so for me the thing that give us meaning so for me the thing that give us meaning so for me the thing it is really sad like I it is really sad like I it is really sad like I half the time I'm excited half the time half the time I'm excited half the time half the time I'm excited half the time I'm uh crying as I'm yes I'm generating I'm uh crying as I'm yes I'm generating I'm uh crying as I'm yes I'm generating code because

code because code because I kind of love programming it's uh it's I kind of love programming it's uh it's I kind of love programming it's uh it's the Act of Creation you you have an idea the Act of Creation you you have an idea the Act of Creation you you have an idea you design it and then you bring it to you design it and then you bring it to you design it and then you bring it to life and it does something especially if life and it does something especially if life and it does something especially if there's some intelligence that it does there's some intelligence that it does there's some intelligence that it does something it doesn't even have to have something it doesn't even have to have something it doesn't even have to have intelligence printing hello world on intelligence printing hello world on intelligence printing hello world on screen you you you made a little machine screen you you you made a little machine screen you you you made a little machine and it comes to life yeah and uh there's and it comes to life yeah and uh there's and it comes to life yeah and uh there's a bunch of tricks you learn along the a bunch of tricks you learn along the a bunch of tricks you learn along the way because you've been doing it for for way because you've been doing it for for way because you've been doing it for for many many years and then for to see AI many many years and then for to see AI many many years and then for to see AI be able to generate all the tricks you be able to generate all the tricks you be able to generate all the tricks you thought were special yeah thought were special yeah thought were special yeah um I don't know it's very um I don't know it's very um I don't know it's very it it it um it's it's scary it's almost painful um it's it's scary it's almost painful um it's it's scary it's almost painful like a loss uh loss of Innocence maybe like a loss uh loss of Innocence maybe like a loss uh loss of Innocence maybe like yeah maybe when when I was younger like yeah maybe when when I was younger like yeah maybe when when I was younger uh I remember before I learned that uh I remember before I learned that uh I remember before I learned that sugar is bad for you you should be on a sugar is bad for you you should be on a sugar is bad for you you should be on a diet diet diet I remember I enjoyed candy deeply in a I remember I enjoyed candy deeply in a I remember I enjoyed candy deeply in a way I just can't anymore that I know is way I just can't anymore that I know is way I just can't anymore that I know is bad for me I enjoyed it unapologetically bad for me I enjoyed it unapologetically bad for me I enjoyed it unapologetically fully just intensely and I just I lost fully just intensely and I just I lost fully just intensely and I just I lost that now I feel like a little bit of that now I feel like a little bit of that now I feel like a little bit of that is lost for me with program or that is lost for me with program or that is lost for me with program or being lost with programming similar as being lost with programming similar as being lost with programming similar as it is for uh the the 3D modeler no it is for uh the the 3D modeler no it is for uh the the 3D modeler no longer being able to really enjoy the longer being able to really enjoy the longer being able to really enjoy the art of modeling uh 3D things for gaming art of modeling uh 3D things for gaming art of modeling uh 3D things for gaming I don't know I don't know what to make I don't know I don't know what to make I don't know I don't know what to make sense of that maybe I would ReDiscover sense of that maybe I would ReDiscover sense of that maybe I would ReDiscover that the true magic of what it means to that the true magic of what it means to that the true magic of what it means to be human is connecting with other humans be human is connecting with other humans be human is connecting with other humans to have conversations like this I don't

to have conversations like this I don't to have conversations like this I don't know to uh to have sex to have to eat know to uh to have sex to have to eat know to uh to have sex to have to eat food to really intensify the value from food to really intensify the value from food to really intensify the value from conscious experiences versus like conscious experiences versus like conscious experiences versus like creating other stuff you're pitching the creating other stuff you're pitching the creating other stuff you're pitching the rebranding again from Homo sapiens rebranding again from Homo sapiens rebranding again from Homo sapiens but the meaningful experiences and just but the meaningful experiences and just but the meaningful experiences and just to inject some optimism in this here so to inject some optimism in this here so to inject some optimism in this here so we don't sound like a bunch of gloomers we don't sound like a bunch of gloomers we don't sound like a bunch of gloomers you know we can totally have our cake you know we can totally have our cake you know we can totally have our cake and eat it you hear a lot of totally and eat it you hear a lot of totally and eat it you hear a lot of totally claims that we can't afford claims that we can't afford claims that we can't afford having more teachers yeah you have to having more teachers yeah you have to having more teachers yeah you have to cut the number of nurses you know that's cut the number of nurses you know that's cut the number of nurses you know that's just nonsense obviously just nonsense obviously just nonsense obviously with anything even with anything even with anything even quite far short of AGI we can quite far short of AGI we can quite far short of AGI we can dramatically improve dramatically improve dramatically improve grow the GDP and produce this wealth of grow the GDP and produce this wealth of grow the GDP and produce this wealth of goods and services it's very easy to goods and services it's very easy to goods and services it's very easy to create a world where everybody is better create a world where everybody is better create a world where everybody is better off than today off than today off than today including the richest people can be including the richest people can be including the richest people can be better off as well right it's not a better off as well right it's not a better off as well right it's not a zero-sum game you know technology again zero-sum game you know technology again zero-sum game you know technology again you can have two countries like Sweden you can have two countries like Sweden you can have two countries like Sweden and Denmark had all these ridiculous and Denmark had all these ridiculous and Denmark had all these ridiculous Wars Century after century Wars Century after century Wars Century after century and uh and uh and uh sometimes that Sweden got a little sometimes that Sweden got a little sometimes that Sweden got a little better off because it got a little bit better off because it got a little bit better off because it got a little bit bigger and then Denmark got a little bit bigger and then Denmark got a little bit bigger and then Denmark got a little bit better off because being a little bit better off because being a little bit better off because being a little bit smaller and and but then we then smaller and and but then we then smaller and and but then we then technology came along and we both got technology came along and we both got technology came along and we both got just dramatically wealthier without just dramatically wealthier without just dramatically wealthier without taking away from anyone else it was just taking away from anyone else it was just taking away from anyone else it was just a total win for everyone a total win for everyone a total win for everyone and uh AI can do that on steroids and uh AI can do that on steroids and uh AI can do that on steroids if you can build safe AGI if you can build safe AGI if you can build safe AGI if you can build super intelligence yeah

if you can build super intelligence yeah if you can build super intelligence yeah basically all the limitations that cause basically all the limitations that cause basically all the limitations that cause harm today can be completely can be harm today can be completely can be harm today can be completely can be completely eliminated right it's a completely eliminated right it's a completely eliminated right it's a wonderful you talk possibility in this wonderful you talk possibility in this wonderful you talk possibility in this this is not sci-fi this is something this is not sci-fi this is something this is not sci-fi this is something which is clearly possible according to which is clearly possible according to which is clearly possible according to laws of physics laws of physics laws of physics and I we can talk about ways of making and I we can talk about ways of making and I we can talk about ways of making it safe also it safe also it safe also um but unfortunately um but unfortunately um but unfortunately that'll only happen if we stare in that that'll only happen if we stare in that that'll only happen if we stare in that direction that's absolutely not the direction that's absolutely not the direction that's absolutely not the default outcome that why default outcome that why default outcome that why income inequality keeps going up that's income inequality keeps going up that's income inequality keeps going up that's why the life expectancy in the US has why the life expectancy in the US has why the life expectancy in the US has been going down now I think it's four been going down now I think it's four been going down now I think it's four years in a row I just read a years in a row I just read a years in a row I just read a heartbreaking study from CDC about how heartbreaking study from CDC about how heartbreaking study from CDC about how something like one-third of all teenage something like one-third of all teenage something like one-third of all teenage girls in the US have been thinking about girls in the US have been thinking about girls in the US have been thinking about suicide you know like suicide you know like suicide you know like those are steps and they're totally the those are steps and they're totally the those are steps and they're totally the wrong direction and and it's important wrong direction and and it's important wrong direction and and it's important to keep our eyes on the prize here that to keep our eyes on the prize here that to keep our eyes on the prize here that we can we have the power now for the we can we have the power now for the we can we have the power now for the first time in the history of our species first time in the history of our species first time in the history of our species to harness artificial intelligence to to harness artificial intelligence to to harness artificial intelligence to help us really flourish help us really flourish help us really flourish and help bring out the best in our and help bring out the best in our and help bring out the best in our Humanity rather than the worst of it Humanity rather than the worst of it Humanity rather than the worst of it to help us have really fulfilling to help us have really fulfilling to help us have really fulfilling experiences that feel truly meaningful experiences that feel truly meaningful experiences that feel truly meaningful and you and I shouldn't sit here and and you and I shouldn't sit here and and you and I shouldn't sit here and dictate the future Generations what they dictate the future Generations what they dictate the future Generations what they will be let them figure it out but let's will be let them figure it out but let's will be let them figure it out but let's give them a chance to live and and not give them a chance to live and and not give them a chance to live and and not foreclose all these possibilities for

foreclose all these possibilities for foreclose all these possibilities for them by just messing things up right them by just messing things up right them by just messing things up right well for that we'll have to solve the AI well for that we'll have to solve the AI well for that we'll have to solve the AI safety problem I just it would be nice safety problem I just it would be nice safety problem I just it would be nice if we can link on exploring that a if we can link on exploring that a if we can link on exploring that a little bit so one interesting way to little bit so one interesting way to little bit so one interesting way to enter that discussion is uh you tweeted enter that discussion is uh you tweeted enter that discussion is uh you tweeted and Elon replied you tweeted let's not and Elon replied you tweeted let's not and Elon replied you tweeted let's not just focus on whether gpt4 we'll do more just focus on whether gpt4 we'll do more just focus on whether gpt4 we'll do more harm or good on the job market but also harm or good on the job market but also harm or good on the job market but also whether it's coding skills will hasten whether it's coding skills will hasten whether it's coding skills will hasten the arrival of super intelligence that's the arrival of super intelligence that's the arrival of super intelligence that's something we've been talking about right something we've been talking about right something we've been talking about right so Elon proposed one thing in the reply so Elon proposed one thing in the reply so Elon proposed one thing in the reply saying maximum truth seeking is my best saying maximum truth seeking is my best saying maximum truth seeking is my best guess for AI safety can you maybe uh guess for AI safety can you maybe uh guess for AI safety can you maybe uh Steel Man the case for this uh since Steel Man the case for this uh since Steel Man the case for this uh since this objective function of Truth and uh this objective function of Truth and uh this objective function of Truth and uh maybe make an argument against it and in maybe make an argument against it and in maybe make an argument against it and in general what uh are your different ideas general what uh are your different ideas general what uh are your different ideas to start approaching the solution to AI to start approaching the solution to AI to start approaching the solution to AI safety I didn't see that reply actually safety I didn't see that reply actually safety I didn't see that reply actually oh interesting all right so but I really oh interesting all right so but I really oh interesting all right so but I really resonate with it because AI is not evil it it caused people AI is not evil it it caused people around the world to hate each other much around the world to hate each other much around the world to hate each other much more more more but that's because we made it in a but that's because we made it in a but that's because we made it in a certain way it's a tool we can use it certain way it's a tool we can use it certain way it's a tool we can use it for great things and bad things and we for great things and bad things and we for great things and bad things and we could just as well have ai systems and could just as well have ai systems and could just as well have ai systems and this is this is part of my vision for this is this is part of my vision for this is this is part of my vision for Success here Success here Success here truth seeking AI that really brings us truth seeking AI that really brings us truth seeking AI that really brings us together again together again together again you know why do people hate each other you know why do people hate each other you know why do people hate each other so much between countries and within so much between countries and within so much between countries and within countries it's because countries it's because countries it's because these have totally different versions of

these have totally different versions of these have totally different versions of the truth right if they all have the same truth that if they all have the same truth that they trusted for good reason because they trusted for good reason because they trusted for good reason because they could check it and verify it and they could check it and verify it and they could check it and verify it and not have to believe in some not have to believe in some not have to believe in some self-proclaimed Authority right self-proclaimed Authority right self-proclaimed Authority right they wouldn't be as nearly as much hate they wouldn't be as nearly as much hate they wouldn't be as nearly as much hate there'd be a lot more understanding there'd be a lot more understanding there'd be a lot more understanding instead and uh instead and uh instead and uh this is this is this is I think something AI can help enormously I think something AI can help enormously I think something AI can help enormously with with with for example for example for example a little baby step in this direction is a little baby step in this direction is a little baby step in this direction is this website called metaculous where People BET and make predictions not for People BET and make predictions not for money but just for their own reputation money but just for their own reputation money but just for their own reputation and it's kind of funny actually you and it's kind of funny actually you and it's kind of funny actually you treat the humans like you treat AIS you treat the humans like you treat AIS you treat the humans like you treat AIS you have a loss function where they get have a loss function where they get have a loss function where they get penalized if they're super confident on penalized if they're super confident on penalized if they're super confident on something and then the opposite happens something and then the opposite happens something and then the opposite happens yeah yeah yeah yeah yeah yeah whereas if you're kind of humble and whereas if you're kind of humble and whereas if you're kind of humble and then you're like I think it's 51 chance then you're like I think it's 51 chance then you're like I think it's 51 chance this is going to happen and then the this is going to happen and then the this is going to happen and then the other happens you don't get penalized other happens you don't get penalized other happens you don't get penalized much and much and much and and what you can see is that some people and what you can see is that some people and what you can see is that some people are much better at predicting than are much better at predicting than are much better at predicting than others they've earned your trusts right others they've earned your trusts right others they've earned your trusts right one project that I'm working on right one project that I'm working on right one project that I'm working on right now is the outgrowth to improve the news now is the outgrowth to improve the news now is the outgrowth to improve the news Foundation together with the metaculous Foundation together with the metaculous Foundation together with the metaculous focuses seeing if we can really scale focuses seeing if we can really scale focuses seeing if we can really scale this up a lot with more powerful AI this up a lot with more powerful AI this up a lot with more powerful AI because I would love it because I would love it because I would love it I would love for there to be like a I would love for there to be like a I would love for there to be like a really powerful truth seeking system really powerful truth seeking system really powerful truth seeking system where where where that is that is that is trustworthy because it keeps being right trustworthy because it keeps being right trustworthy because it keeps being right about stuff about stuff about stuff and people who come to it and and people who come to it and and people who come to it and maybe look at its latest trust ranking

maybe look at its latest trust ranking maybe look at its latest trust ranking of different pundits and newspapers Etc of different pundits and newspapers Etc of different pundits and newspapers Etc if they want to know why some someone if they want to know why some someone if they want to know why some someone got a low score they can click on it and got a low score they can click on it and got a low score they can click on it and see all that predictions that they see all that predictions that they see all that predictions that they actually made and how they turned out actually made and how they turned out actually made and how they turned out you know you know you know this is how we do it in science you this is how we do it in science you this is how we do it in science you trust scientists like Einstein who said trust scientists like Einstein who said trust scientists like Einstein who said something everybody thought was something everybody thought was something everybody thought was and turned out to be right and turned out to be right and turned out to be right you get a lot of trust Point than he did you get a lot of trust Point than he did you get a lot of trust Point than he did it multiple times even I think AI has the power to really heal I think AI has the power to really heal a lot of the the Rifts we're seeing by a lot of the the Rifts we're seeing by a lot of the the Rifts we're seeing by creating Trust Systems creating Trust Systems creating Trust Systems it has to get away from this idea today it has to get away from this idea today it has to get away from this idea today with some fact-checking sites which with some fact-checking sites which with some fact-checking sites which might themselves have an agenda and you might themselves have an agenda and you might themselves have an agenda and you just trust it because of its reputation just trust it because of its reputation just trust it because of its reputation to to to you want to have it so this so these you want to have it so this so these you want to have it so this so these sort of systems they are in their trust sort of systems they are in their trust sort of systems they are in their trust and that's completely transparent and that's completely transparent and that's completely transparent this I think would actually help a lot this I think would actually help a lot this I think would actually help a lot uh that you know I think help heal the uh that you know I think help heal the uh that you know I think help heal the very dysfunctional conversation that very dysfunctional conversation that very dysfunctional conversation that Humanity has about how it's going to Humanity has about how it's going to Humanity has about how it's going to deal with all its deal with all its deal with all its biggest challenges in in the world today biggest challenges in in the world today biggest challenges in in the world today and then uh and then uh and then uh enter on the technical side you know enter on the technical side you know enter on the technical side you know another common sort of Gloom uh another common sort of Gloom uh another common sort of Gloom uh comment I get from people are saying comment I get from people are saying comment I get from people are saying we're just screwed there's no hope we're just screwed there's no hope we're just screwed there's no hope is well things like gpt4 are way too is well things like gpt4 are way too is well things like gpt4 are way too complicated for a human to ever complicated for a human to ever complicated for a human to ever understand and prove that they can be understand and prove that they can be understand and prove that they can be trustworthy they're forgetting that AI trustworthy they're forgetting that AI trustworthy they're forgetting that AI can help us prove that things work right can help us prove that things work right can help us prove that things work right yeah and and there's this very

yeah and and there's this very yeah and and there's this very fundamental fact that in math it's much fundamental fact that in math it's much fundamental fact that in math it's much harder to come up with a proof that it harder to come up with a proof that it harder to come up with a proof that it is to verify that the proof is correct is to verify that the proof is correct is to verify that the proof is correct you can actually write a little proof you can actually write a little proof you can actually write a little proof checking code it's quite short checking code it's quite short checking code it's quite short but you can assume and understand and but you can assume and understand and but you can assume and understand and then it can check the most monstrously then it can check the most monstrously then it can check the most monstrously long proof ever Drake generated even by long proof ever Drake generated even by long proof ever Drake generated even by a computer and say yeah this is valid a computer and say yeah this is valid a computer and say yeah this is valid so so right now so so right now so so right now we we have we we have we we have this uh this uh this uh this approach with virus checking this approach with virus checking this approach with virus checking software that it looks to see if there's software that it looks to see if there's software that it looks to see if there's something you should not trust it and if something you should not trust it and if something you should not trust it and if it can prove to itself that you should it can prove to itself that you should it can prove to itself that you should not trust that code it warns you right not trust that code it warns you right not trust that code it warns you right what if you flip this around what if you flip this around what if you flip this around and this is an idea I should give credit and this is an idea I should give credit and this is an idea I should give credit to Steve I'm 100 for so that it will to Steve I'm 100 for so that it will to Steve I'm 100 for so that it will only run the code if it can prove only run the code if it can prove only run the code if it can prove instead of not running it if it can instead of not running it if it can instead of not running it if it can prove that it's not trustworthy if it prove that it's not trustworthy if it prove that it's not trustworthy if it will only run and if it can prove that will only run and if it can prove that will only run and if it can prove that it's trustworthy so it asks the code it it's trustworthy so it asks the code it it's trustworthy so it asks the code it prove to me that you're going to do what prove to me that you're going to do what prove to me that you're going to do what you say you're going to do you say you're going to do you say you're going to do and and it gives you this proof and and it gives you this proof and and it gives you this proof and you a little proof that you can and you a little proof that you can and you a little proof that you can check it now you can actually trust check it now you can actually trust check it now you can actually trust an AI That's much more intelligent than an AI That's much more intelligent than an AI That's much more intelligent than you are right you are right you are right because you it's it's problem to come up because you it's it's problem to come up because you it's it's problem to come up with this proof that you could never with this proof that you could never with this proof that you could never have found that you should trust it so have found that you should trust it so have found that you should trust it so this is the interesting point I agree this is the interesting point I agree this is the interesting point I agree with you but this is where Eleazar with you but this is where Eleazar with you but this is where Eleazar yakowski might disagree with you his yakowski might disagree with you his yakowski might disagree with you his claim not with you but with this idea

claim not with you but with this idea claim not with you but with this idea is his claim is super intelligent AI is his claim is super intelligent AI is his claim is super intelligent AI would be able to know how to lie to you would be able to know how to lie to you would be able to know how to lie to you with such a proof with such a proof with such a proof I'll Delight to you and give me a proof I'll Delight to you and give me a proof I'll Delight to you and give me a proof that I'm gonna think is correct yeah so that I'm gonna think is correct yeah so that I'm gonna think is correct yeah so but it's not me it's lying to that's the but it's not me it's lying to that's the but it's not me it's lying to that's the trick my proof checker trick my proof checker trick my proof checker which is a piece of code so his general which is a piece of code so his general which is a piece of code so his general idea is a super intelligent system idea is a super intelligent system idea is a super intelligent system can lie to a dumber proof checker can lie to a dumber proof checker can lie to a dumber proof checker so you're going to have as a system so you're going to have as a system so you're going to have as a system becomes more and more intelligent becomes more and more intelligent becomes more and more intelligent there's going to be a threshold there's going to be a threshold there's going to be a threshold or a super intelligent system will be or a super intelligent system will be or a super intelligent system will be able to effectively lie to a slightly able to effectively lie to a slightly able to effectively lie to a slightly Dumber AGI system uh like there's a Dumber AGI system uh like there's a Dumber AGI system uh like there's a threat like he really focuses on this threat like he really focuses on this threat like he really focuses on this weak AGI the strong AGI jump with a weak AGI the strong AGI jump with a weak AGI the strong AGI jump with a strong AGI can make all the weak age you strong AGI can make all the weak age you strong AGI can make all the weak age you guys think that it's just one of them guys think that it's just one of them guys think that it's just one of them but it's no longer that but it's no longer that but it's no longer that and that leap is when it runs away yeah and that leap is when it runs away yeah and that leap is when it runs away yeah I I don't buy that argument I think no I I don't buy that argument I think no I I don't buy that argument I think no matter how super intelligent an AI is matter how super intelligent an AI is matter how super intelligent an AI is it's never going to be able to prove to it's never going to be able to prove to it's never going to be able to prove to me that they're only finitely many me that they're only finitely many me that they're only finitely many primes for example it can try to snow Me by making up all it can try to snow Me by making up all sorts of the new weird rules of sorts of the new weird rules of sorts of the new weird rules of deduction deduction deduction that and say trust me you know that and say trust me you know that and say trust me you know the way your proof jacket work is too the way your proof jacket work is too the way your proof jacket work is too limited and we have this new hyper map limited and we have this new hyper map limited and we have this new hyper map and it's true and it's true and it's true um but then I would I would just uh take um but then I would I would just uh take um but then I would I would just uh take the attitude okay I'm gonna forfeit some the attitude okay I'm gonna forfeit some the attitude okay I'm gonna forfeit some of these this supposedly super cool

of these this supposedly super cool of these this supposedly super cool technology so I'm only going to go with technology so I'm only going to go with technology so I'm only going to go with the ones that I can prove in my own the ones that I can prove in my own the ones that I can prove in my own trusted proof gesture then I don't I trusted proof gesture then I don't I trusted proof gesture then I don't I think it's fine there's still of course think it's fine there's still of course think it's fine there's still of course this is not something anyone has this is not something anyone has this is not something anyone has successfully implemented at this point successfully implemented at this point successfully implemented at this point but I think it I just give it as an but I think it I just give it as an but I think it I just give it as an example of hope we don't have to do all example of hope we don't have to do all example of hope we don't have to do all the work ourselves right this is exactly the work ourselves right this is exactly the work ourselves right this is exactly the sort of very boring and tedious task the sort of very boring and tedious task the sort of very boring and tedious task that is perfect to Outsource to an AI that is perfect to Outsource to an AI that is perfect to Outsource to an AI and this is a way in which less powerful and this is a way in which less powerful and this is a way in which less powerful and less intelligent agents like us can and less intelligent agents like us can and less intelligent agents like us can actually actually actually continue to control and Trust more continue to control and Trust more continue to control and Trust more powerful ones so build AGI systems that powerful ones so build AGI systems that powerful ones so build AGI systems that help us defend against other AGI systems help us defend against other AGI systems help us defend against other AGI systems well for starters well for starters well for starters begin with a simple problem of just begin with a simple problem of just begin with a simple problem of just making sure that the system that you own making sure that the system that you own making sure that the system that you own or that's supposed to be loyal to you or that's supposed to be loyal to you or that's supposed to be loyal to you has to prove to itself that it's always has to prove to itself that it's always has to prove to itself that it's always going to do the things that you actually going to do the things that you actually going to do the things that you actually wanted to do right and if it can't prove wanted to do right and if it can't prove wanted to do right and if it can't prove it maybe it's still going to do it but it maybe it's still going to do it but it maybe it's still going to do it but you won't run it so you just forfeit you won't run it so you just forfeit you won't run it so you just forfeit some aspects of all the cool things AI some aspects of all the cool things AI some aspects of all the cool things AI can do I I bet you dollars of donuts you can do I I bet you dollars of donuts you can do I I bet you dollars of donuts you can still do some incredibly cool stuff can still do some incredibly cool stuff can still do some incredibly cool stuff for you yeah there are other things too for you yeah there are other things too for you yeah there are other things too that we shouldn't sweep under the rug that we shouldn't sweep under the rug that we shouldn't sweep under the rug like not every human agrees on exactly like not every human agrees on exactly like not every human agrees on exactly where what direction we should go with where what direction we should go with where what direction we should go with Humanity right yes and you've talked a Humanity right yes and you've talked a Humanity right yes and you've talked a lot about geopolitical things on this on lot about geopolitical things on this on lot about geopolitical things on this on on your podcast to this effect you know on your podcast to this effect you know on your podcast to this effect you know but but but I think that shouldn't distract us from I think that shouldn't distract us from I think that shouldn't distract us from the fact that there are actually a lot the fact that there are actually a lot the fact that there are actually a lot of things that everybody in the world of things that everybody in the world of things that everybody in the world virtually agrees on that hey you know virtually agrees on that hey you know virtually agrees on that hey you know like having no humans on the planet

in a in the near future let's not do in a in the near future let's not do that right you looked at something like that right you looked at something like that right you looked at something like the United Nations sustainable the United Nations sustainable the United Nations sustainable development goals some of them are development goals some of them are development goals some of them are required required required ambitious and uh basically all the ambitious and uh basically all the ambitious and uh basically all the countries agree U.S China Russia Ukraine countries agree U.S China Russia Ukraine countries agree U.S China Russia Ukraine they all agree so instead of quibbling they all agree so instead of quibbling they all agree so instead of quibbling about the little things we don't agree about the little things we don't agree about the little things we don't agree on let's start with the things we do on let's start with the things we do on let's start with the things we do with Beyond and and get them done with Beyond and and get them done with Beyond and and get them done instead of being so distracted by all instead of being so distracted by all instead of being so distracted by all these things we disagree on these things we disagree on these things we disagree on that warlock wins because that warlock wins because that warlock wins because frankly frankly frankly molok going wild now it feels like a war molok going wild now it feels like a war molok going wild now it feels like a war on Life playing out in front of our eyes on Life playing out in front of our eyes on Life playing out in front of our eyes if you if you just look at it if you if you just look at it if you if you just look at it from space you know we're on this planet from space you know we're on this planet from space you know we're on this planet beautiful vibrant ecosystem now we start beautiful vibrant ecosystem now we start beautiful vibrant ecosystem now we start chopping down chopping down chopping down big parts of it even though nobody most big parts of it even though nobody most big parts of it even though nobody most people thought that was a bad idea people thought that was a bad idea people thought that was a bad idea always start doing ocean acidification always start doing ocean acidification always start doing ocean acidification wiping out all sorts of species oh now wiping out all sorts of species oh now wiping out all sorts of species oh now we have all these close calls we almost we have all these close calls we almost we have all these close calls we almost had a nuclear war and we're replacing had a nuclear war and we're replacing had a nuclear war and we're replacing more and more of the biosphere with more and more of the biosphere with more and more of the biosphere with non-living things we're also replacing non-living things we're also replacing non-living things we're also replacing in our social lives a lot of the things in our social lives a lot of the things in our social lives a lot of the things which were so valuable to humanity a lot which were so valuable to humanity a lot which were so valuable to humanity a lot of social interactions now are replaced of social interactions now are replaced of social interactions now are replaced by people staring into their rectangles by people staring into their rectangles by people staring into their rectangles right and I right and I right and I I'm not a psychologist I'm out of my I'm not a psychologist I'm out of my I'm not a psychologist I'm out of my depth here but I suspect that part of depth here but I suspect that part of depth here but I suspect that part of the reason why teen suicide and suicide the reason why teen suicide and suicide the reason why teen suicide and suicide in general in the US is at

in general in the US is at in general in the US is at record-breaking levels is actually record-breaking levels is actually record-breaking levels is actually caused by caused by caused by again and so AI Technologies and social again and so AI Technologies and social again and so AI Technologies and social media making people spend less time with media making people spend less time with media making people spend less time with with actual and actually just human with actual and actually just human with actual and actually just human interaction we've all seen interaction we've all seen interaction we've all seen a bunch of good looking people in a bunch of good looking people in a bunch of good looking people in restaurants restaurants restaurants into the rectangles instead of looking into the rectangles instead of looking into the rectangles instead of looking at each other's eyes right at each other's eyes right at each other's eyes right so that's also a part of the war in life so that's also a part of the war in life so that's also a part of the war in life that that we're we're replacing so many really life-affirming things really life-affirming things by technology we're we're putting by technology we're we're putting by technology we're we're putting technology between us technology between us technology between us the the technology that was supposed to the the technology that was supposed to the the technology that was supposed to connect us is actually distancing us connect us is actually distancing us connect us is actually distancing us ourselves from each other and um ourselves from each other and um ourselves from each other and um and then we're giving ever more power to and then we're giving ever more power to and then we're giving ever more power to things which are not alive these large things which are not alive these large things which are not alive these large corporations are not living things right corporations are not living things right corporations are not living things right they're just maximizing profit they're just maximizing profit they're just maximizing profit there there there I want to win them more in life I I I want to win them more in life I I I want to win them more in life I I think we humans together with all our think we humans together with all our think we humans together with all our fellow living things on this planet will fellow living things on this planet will fellow living things on this planet will be better off if we can be better off if we can be better off if we can remain in control over the non-living remain in control over the non-living remain in control over the non-living things and make sure that they they work things and make sure that they they work things and make sure that they they work for us I really think it can be done for us I really think it can be done for us I really think it can be done can you just Linger on this uh maybe can you just Linger on this uh maybe can you just Linger on this uh maybe high level philosophical disagreement high level philosophical disagreement high level philosophical disagreement with Eliezer yadkowski with Eliezer yadkowski with Eliezer yadkowski bye in this the hope you're stating so bye in this the hope you're stating so bye in this the hope you're stating so he is very sure he is very sure he is very sure he puts a very high probability

he puts a very high probability he puts a very high probability very close to one depending on the day very close to one depending on the day very close to one depending on the day he puts it at one he puts it at one he puts it at one that AI is going to kill humans that AI is going to kill humans that AI is going to kill humans um um um that there's just he does not see a that there's just he does not see a that there's just he does not see a trajectory which it doesn't end up with trajectory which it doesn't end up with trajectory which it doesn't end up with that conclusion what uh which trajectory that conclusion what uh which trajectory that conclusion what uh which trajectory do you see that doesn't end up there and do you see that doesn't end up there and do you see that doesn't end up there and maybe can you maybe can you maybe can you can you see the point he's making can you see the point he's making can you see the point he's making and and can you also see a way out and and can you also see a way out and and can you also see a way out mm-hmm mm-hmm mm-hmm first of all I tremendously respect first of all I tremendously respect first of all I tremendously respect Elias yukowski and his his thinking Elias yukowski and his his thinking Elias yukowski and his his thinking second I do share his view that there's second I do share his view that there's second I do share his view that there's a pretty large chance that we're not a pretty large chance that we're not a pretty large chance that we're not going to make it as humans there won't going to make it as humans there won't going to make it as humans there won't be any humans on the planet be any humans on the planet be any humans on the planet and then not the distance future and and then not the distance future and and then not the distance future and that makes me very sad you know we just that makes me very sad you know we just that makes me very sad you know we just had a little baby and I keep asking had a little baby and I keep asking had a little baby and I keep asking myself you know is um um um how old is he even gonna get you know how old is he even gonna get you know how old is he even gonna get you know and and um I ask myself hey and and um I ask myself hey and and um I ask myself hey it feels I said to my wife recently it it feels I said to my wife recently it it feels I said to my wife recently it feels a little bit like I was just feels a little bit like I was just feels a little bit like I was just diagnosed with some sort of um diagnosed with some sort of um diagnosed with some sort of um cancer which has some you know risk of cancer which has some you know risk of cancer which has some you know risk of of dying from in some risk of surviving of dying from in some risk of surviving of dying from in some risk of surviving you know you know you know uh uh uh except this is the kind of cancer which except this is the kind of cancer which except this is the kind of cancer which will kill all of humanity so I will kill all of humanity so I will kill all of humanity so I completely take seriously his completely take seriously his completely take seriously his um his concerns um his concerns um his concerns I think um I think um I think um but I don't absolutely don't think it's but I don't absolutely don't think it's but I don't absolutely don't think it's hopeless I think um there is a there is

hopeless I think um there is a there is hopeless I think um there is a there is a a a first of all a lot of momentum now first of all a lot of momentum now first of all a lot of momentum now for the first time actually since the for the first time actually since the for the first time actually since the many many years that have passed since many many years that have passed since many many years that have passed since since I and many others started warning since I and many others started warning since I and many others started warning about this I feel about this I feel about this I feel most people are getting it now most people are getting it now most people are getting it now I I I I I uh I I uh I I uh it's just talking it's just talking it's just talking to this guy in the gas station they were to this guy in the gas station they were to this guy in the gas station they were a house the other day my a house the other day my a house the other day my and he's like and he's like and he's like I think we're getting replaced I think we're getting replaced I think we're getting replaced and then I think in it so that's and then I think in it so that's and then I think in it so that's positive that they're finally we're positive that they're finally we're positive that they're finally we're finally seeing this reaction which is finally seeing this reaction which is finally seeing this reaction which is the first step towards solving the the first step towards solving the the first step towards solving the problem uh second uh I really think that problem uh second uh I really think that problem uh second uh I really think that this this vision of only running ai's this this vision of only running ai's this this vision of only running ai's really if the stakes are really high really if the stakes are really high really if the stakes are really high they can prove to us that they're safe they can prove to us that they're safe they can prove to us that they're safe it's really just virus checking in it's really just virus checking in it's really just virus checking in Reverse again I I think it's Reverse again I I think it's Reverse again I I think it's scientifically doable scientifically doable scientifically doable I don't think it's hopeless um I don't think it's hopeless um I don't think it's hopeless um we might have to Forfeit some of the we might have to Forfeit some of the we might have to Forfeit some of the technology that we could get if we were technology that we could get if we were technology that we could get if we were putting Blind Faith in our AIS but we're putting Blind Faith in our AIS but we're putting Blind Faith in our AIS but we're still going to get amazing stuff do you still going to get amazing stuff do you still going to get amazing stuff do you envision a process with the proof envision a process with the proof envision a process with the proof Checker like something like gpt4 GPT 5 Checker like something like gpt4 GPT 5 Checker like something like gpt4 GPT 5 will go through a process no rigorous no will go through a process no rigorous no will go through a process no rigorous no no I think it's hopeless that's like no I think it's hopeless that's like no I think it's hopeless that's like trying to prove there about five trying to prove there about five trying to prove there about five spaghetti spaghetti spaghetti okay what I think well how the the whole okay what I think well how the the whole okay what I think well how the the whole the vision I have for success is instead the vision I have for success is instead the vision I have for success is instead that you know just like we human beings that you know just like we human beings that you know just like we human beings were able to look at our brains and and were able to look at our brains and and were able to look at our brains and and they still out the key knowledge Galileo they still out the key knowledge Galileo they still out the key knowledge Galileo when his dad threw him an apple when he

when his dad threw him an apple when he when his dad threw him an apple when he was a kid he was able to catch it was a kid he was able to catch it was a kid he was able to catch it because his brain could in this funny because his brain could in this funny because his brain could in this funny spaghetti kind of way you know predict spaghetti kind of way you know predict spaghetti kind of way you know predict how parabolas are going to move his how parabolas are going to move his how parabolas are going to move his Kahneman system one right but then he Kahneman system one right but then he Kahneman system one right but then he got older and it's like wait got older and it's like wait got older and it's like wait this is a parabola it's it's y equals x this is a parabola it's it's y equals x this is a parabola it's it's y equals x squared I can distill this knowledge out squared I can distill this knowledge out squared I can distill this knowledge out and today you can easily program it into and today you can easily program it into and today you can easily program it into a computer and it can simulate not just a computer and it can simulate not just a computer and it can simulate not just that but how to get to Mars and so on that but how to get to Mars and so on that but how to get to Mars and so on right I Envision a similar process where right I Envision a similar process where right I Envision a similar process where we use the the amazing learning power of we use the the amazing learning power of we use the the amazing learning power of neural networks neural networks neural networks to discover the knowledge in the first to discover the knowledge in the first to discover the knowledge in the first place but we don't stop with a black box place but we don't stop with a black box place but we don't stop with a black box and and use that we then do a second and and use that we then do a second and and use that we then do a second round of AI where we use automated round of AI where we use automated round of AI where we use automated systems to extract out the knowledge and systems to extract out the knowledge and systems to extract out the knowledge and see what is it what are the insights see what is it what are the insights see what is it what are the insights it's had okay and it's it's had okay and it's it's had okay and it's and then we we put that knowledge into a and then we we put that knowledge into a and then we we put that knowledge into a completely different kind of um completely different kind of um completely different kind of um architecture programming language or architecture programming language or architecture programming language or whatever that's that's made in a way whatever that's that's made in a way whatever that's that's made in a way that it can be both really efficient and that it can be both really efficient and that it can be both really efficient and also also also is more amenable to very formal is more amenable to very formal is more amenable to very formal verification verification verification that's that's my vision I'm not saying that's that's my vision I'm not saying that's that's my vision I'm not saying sitting here saying I'm confident 100 sitting here saying I'm confident 100 sitting here saying I'm confident 100 sure that it's going to work you know sure that it's going to work you know sure that it's going to work you know but I don't think it's a chance it's but I don't think it's a chance it's but I don't think it's a chance it's certainly not zero either and it will certainly not zero either and it will certainly not zero either and it will certainly be possible to do for a lot of certainly be possible to do for a lot of certainly be possible to do for a lot of really cool AI applications that we're really cool AI applications that we're really cool AI applications that we're not using now so we can have a lot of not using now so we can have a lot of not using now so we can have a lot of the fun that we're excited about the fun that we're excited about the fun that we're excited about if we if we do this we're going to need if we if we do this we're going to need if we if we do this we're going to need a little bit of time a little bit of time a little bit of time that's why it's good to pause and put in that's why it's good to pause and put in that's why it's good to pause and put in place requirements

one more thing also I I think one more thing also I I think you know someone might think well zero you know someone might think well zero you know someone might think well zero percent chance we're gonna survive let's percent chance we're gonna survive let's percent chance we're gonna survive let's just give up right that's very dangerous because there's no more guaranteed way because there's no more guaranteed way it's a fail than to convince yourself it's a fail than to convince yourself it's a fail than to convince yourself that it's impossible and not to try that it's impossible and not to try that it's impossible and not to try you know any if you you know when you you know any if you you know when you you know any if you you know when you study history in military history the study history in military history the study history in military history the first thing you learn is that that's first thing you learn is that that's first thing you learn is that that's how you do psychological warfare you how you do psychological warfare you how you do psychological warfare you persuade the other side that it's persuade the other side that it's persuade the other side that it's hopeless so they don't even fight hopeless so they don't even fight hopeless so they don't even fight and then then of course you win right and then then of course you win right and then then of course you win right let's not do this uh psychological let's not do this uh psychological let's not do this uh psychological warfare on ourselves and say there's a warfare on ourselves and say there's a warfare on ourselves and say there's a hundred percent probability we're all hundred percent probability we're all hundred percent probability we're all gonna we're all screwed anyway it's sadly I I do get that a little bit it's sadly I I do get that a little bit sometimes from from uh some young people sometimes from from uh some young people sometimes from from uh some young people who are like so convinced that we're all who are like so convinced that we're all who are like so convinced that we're all screwed that they're like I'm just gonna screwed that they're like I'm just gonna screwed that they're like I'm just gonna play game play computer games and do play game play computer games and do play game play computer games and do drugs and because we're screwed anyway drugs and because we're screwed anyway drugs and because we're screwed anyway right right right it's important it's important it's important keep the hope alive because it actually keep the hope alive because it actually keep the hope alive because it actually has a causal impact and making it makes has a causal impact and making it makes has a causal impact and making it makes it more likely that we're going to it more likely that we're going to it more likely that we're going to succeed it seems like the people that succeed it seems like the people that succeed it seems like the people that actually build solutions to the problem actually build solutions to the problem actually build solutions to the problem seemingly impossible to solve problems seemingly impossible to solve problems seemingly impossible to solve problems are the ones that believe yeah they're are the ones that believe yeah they're are the ones that believe yeah they're the ones who are the optimists yeah and the ones who are the optimists yeah and the ones who are the optimists yeah and it's like uh it seems like there's some it's like uh it seems like there's some it's like uh it seems like there's some fundamental law to the universe where fundamental law to the universe where fundamental law to the universe where fake it till you make it kind of works fake it till you make it kind of works fake it till you make it kind of works like believe it's possible and it like believe it's possible and it like believe it's possible and it becomes possible yeah was it Henry Ford becomes possible yeah was it Henry Ford becomes possible yeah was it Henry Ford who said that

who said that who said that if you can if you tell yourself that if you can if you tell yourself that if you can if you tell yourself that it's impossible it is it's impossible it is it's impossible it is so let's not make that mistake yeah and so let's not make that mistake yeah and so let's not make that mistake yeah and this is a big mistake Society is making this is a big mistake Society is making this is a big mistake Society is making or I think all in all everybody's so or I think all in all everybody's so or I think all in all everybody's so gloomy and the media are also very gloomy and the media are also very gloomy and the media are also very biased towards if it bleeds it bleeds biased towards if it bleeds it bleeds biased towards if it bleeds it bleeds and Bloom and doom right so and Bloom and doom right so and Bloom and doom right so um um um most most most visions of the future we have or or a visions of the future we have or or a visions of the future we have or or a dystopian which really demotivates dystopian which really demotivates dystopian which really demotivates people we want a really really focus on people we want a really really focus on people we want a really really focus on the upside also to give people the the upside also to give people the the upside also to give people the willingness to fight for it willingness to fight for it willingness to fight for it and um for AI you and I mostly talked about for AI you and I mostly talked about Gloom here again but let's not remember Gloom here again but let's not remember Gloom here again but let's not remember not forget that you know not forget that you know not forget that you know we have probably both lost we have probably both lost we have probably both lost someone we really cared about some someone we really cared about some someone we really cared about some disease that we were told was incurable disease that we were told was incurable disease that we were told was incurable well it's not there's no law in physics well it's not there's no law in physics well it's not there's no law in physics saying they have to die of that cancer saying they have to die of that cancer saying they have to die of that cancer or whatever of course you can cure it or whatever of course you can cure it or whatever of course you can cure it and there's so many other things where and there's so many other things where and there's so many other things where that we with a human intelligence have that we with a human intelligence have that we with a human intelligence have also failed to solve on this planet also failed to solve on this planet also failed to solve on this planet which AI could also very much help us which AI could also very much help us which AI could also very much help us with right so if we can get this right with right so if we can get this right with right so if we can get this right just be a little more chill and slow just be a little more chill and slow just be a little more chill and slow down a little bit so we get it right down a little bit so we get it right down a little bit so we get it right it's mind-blowing how awesome our future it's mind-blowing how awesome our future it's mind-blowing how awesome our future can be right we talked a lot about stuff can be right we talked a lot about stuff can be right we talked a lot about stuff on Earth it can be great on Earth it can be great on Earth it can be great but even if you really get ambitious and but even if you really get ambitious and but even if you really get ambitious and look up at the skies right there's no look up at the skies right there's no look up at the skies right there's no reason we have to be stuck on this reason we have to be stuck on this reason we have to be stuck on this planet for the rest of um planet for the rest of um planet for the rest of um the remain for billions of years to come

the remain for billions of years to come the remain for billions of years to come we totally understand now that laws of we totally understand now that laws of we totally understand now that laws of physics let life spread out into space physics let life spread out into space physics let life spread out into space to other solar systems to other galaxies to other solar systems to other galaxies to other solar systems to other galaxies and flourish for billions of billions of and flourish for billions of billions of and flourish for billions of billions of years years years and this to me is a very very hopeful and this to me is a very very hopeful and this to me is a very very hopeful vision vision vision that really motivates me to to fight that really motivates me to to fight that really motivates me to to fight them coming back to in the end something them coming back to in the end something them coming back to in the end something you talked about again you know this the you talked about again you know this the you talked about again you know this the struggle how the human struggle is one struggle how the human struggle is one struggle how the human struggle is one of the things that also really gives of the things that also really gives of the things that also really gives meaning to our lives if there's ever meaning to our lives if there's ever meaning to our lives if there's ever been an epic struggle been an epic struggle been an epic struggle this is it and isn't it even more epic this is it and isn't it even more epic this is it and isn't it even more epic if you're the underdog if you're the underdog if you're the underdog if most people are telling you this is if most people are telling you this is if most people are telling you this is gonna fail it's impossible right and you gonna fail it's impossible right and you gonna fail it's impossible right and you persist persist persist and you succeed and you succeed and you succeed right and that's what we can do together right and that's what we can do together right and that's what we can do together as a species on this one a lot of as a species on this one a lot of as a species on this one a lot of pundits are ready to count this out pundits are ready to count this out pundits are ready to count this out both in the battle to keep AI safe and both in the battle to keep AI safe and both in the battle to keep AI safe and becoming a multiple planetary species becoming a multiple planetary species becoming a multiple planetary species yeah and they're they're the same yeah and they're they're the same yeah and they're they're the same challenge if we can keep AI safe that's challenge if we can keep AI safe that's challenge if we can keep AI safe that's how we're gonna get multi-planatory very how we're gonna get multi-planatory very how we're gonna get multi-planatory very efficiently efficiently efficiently I had some sort of technical questions I had some sort of technical questions I had some sort of technical questions about how to get it right so one idea about how to get it right so one idea about how to get it right so one idea uh that I'm not even sure what the right uh that I'm not even sure what the right uh that I'm not even sure what the right answer is to is uh should systems like answer is to is uh should systems like answer is to is uh should systems like gpt4 be open sourced in the whole arm gpt4 be open sourced in the whole arm gpt4 be open sourced in the whole arm part can you make the can you see the part can you make the can you see the part can you make the can you see the case for either case for either case for either I think the answer right now is no I think the answer right now is no I think the answer right now is no I think the answer early on was yes I think the answer early on was yes I think the answer early on was yes so we could bring in the all the

so we could bring in the all the so we could bring in the all the wonderful create the thought process of wonderful create the thought process of wonderful create the thought process of everybody on this but asking should we everybody on this but asking should we everybody on this but asking should we open source gpt4 now is just the same as open source gpt4 now is just the same as open source gpt4 now is just the same as if you say well is it good should we if you say well is it good should we if you say well is it good should we open source open source open source um um um you know how to build really small you know how to build really small you know how to build really small nuclear weapons should we open source nuclear weapons should we open source nuclear weapons should we open source how to make bio weapons how to make bio weapons how to make bio weapons to the open source how to make um to the open source how to make um to the open source how to make um a new virus that kills 90 of everybody a new virus that kills 90 of everybody a new virus that kills 90 of everybody who gets it of course we shouldn't who gets it of course we shouldn't who gets it of course we shouldn't so it's already that powerful so it's already that powerful so it's already that powerful it's already that powerful that we have it's already that powerful that we have it's already that powerful that we have to respect the power of the systems to respect the power of the systems to respect the power of the systems we've built we've built we've built the knowledge that you get the knowledge that you get the knowledge that you get um um um from open sourcing everything we do now from open sourcing everything we do now from open sourcing everything we do now might very well be powerful enough that might very well be powerful enough that might very well be powerful enough that people looking at that people looking at that people looking at that can use it to build the things that are can use it to build the things that are can use it to build the things that are really threatening again let's get it really threatening again let's get it really threatening again let's get it remember open AI is remember open AI is remember open AI is gpt4 is a baby AI maybe a sort of baby gpt4 is a baby AI maybe a sort of baby gpt4 is a baby AI maybe a sort of baby Proto almost a little bit AGI according Proto almost a little bit AGI according Proto almost a little bit AGI according to what Microsoft's recent paper said to what Microsoft's recent paper said to what Microsoft's recent paper said right right right it's not that they were scared of what it's not that they were scared of what it's not that they were scared of what we're scared about is people taking that we're scared about is people taking that we're scared about is people taking that who are who are who are who might be a lot less responsible than who might be a lot less responsible than who might be a lot less responsible than the company that made it right and uh the company that made it right and uh the company that made it right and uh just going to town with it that's why just going to town with it that's why just going to town with it that's why we want to it's it's an information we want to it's it's an information we want to it's it's an information Hazard there are many things which um Hazard there are many things which um Hazard there are many things which um you know are not open sourced right now you know are not open sourced right now you know are not open sourced right now in society for very good reason in society for very good reason in society for very good reason like how how do you make

like how how do you make like how how do you make certain kind of very powerful toxins out certain kind of very powerful toxins out certain kind of very powerful toxins out of stuff you can buy at Home Depot you of stuff you can buy at Home Depot you of stuff you can buy at Home Depot you know you know you don't open source know you know you don't open source know you know you don't open source those things for a reason those things for a reason those things for a reason and this is really no different and this is really no different and this is really no different so uh what I'm saying that I have to say so uh what I'm saying that I have to say so uh what I'm saying that I have to say it feels in a bit weird but in a way a it feels in a bit weird but in a way a it feels in a bit weird but in a way a bit weird to say it because MIT is like bit weird to say it because MIT is like bit weird to say it because MIT is like the Cradle of the open source movement the Cradle of the open source movement the Cradle of the open source movement and I love open source in general power and I love open source in general power and I love open source in general power to the people let's say um to the people let's say um to the people let's say um but um there's always gonna be some but um there's always gonna be some but um there's always gonna be some stuff that you don't open source and you stuff that you don't open source and you stuff that you don't open source and you know it's just like you don't open know it's just like you don't open know it's just like you don't open source so we have a three-month-old baby source so we have a three-month-old baby source so we have a three-month-old baby right when he gets a little bit older right when he gets a little bit older right when he gets a little bit older we're not going to open source to him we're not going to open source to him we're not going to open source to him all the most dangerous things he could all the most dangerous things he could all the most dangerous things he could do in the house yeah right do in the house yeah right do in the house yeah right but it does it's a weird feeling because but it does it's a weird feeling because but it does it's a weird feeling because this is one of the first moments in this is one of the first moments in this is one of the first moments in history where so there's a strong case history where so there's a strong case history where so there's a strong case to be made not to open source software to be made not to open source software to be made not to open source software this is when the software has become this is when the software has become this is when the software has become yeah too dangerous yeah but it's not the yeah too dangerous yeah but it's not the yeah too dangerous yeah but it's not the first time that we didn't want to open first time that we didn't want to open first time that we didn't want to open source a technology technology yeah is there something to be said about how is there something to be said about how to get the release of such systems right to get the release of such systems right to get the release of such systems right like GPT 4 and gpt5 like GPT 4 and gpt5 like GPT 4 and gpt5 so open AI went through a pretty so open AI went through a pretty so open AI went through a pretty rigorous effort for several months you rigorous effort for several months you rigorous effort for several months you could say it could be longer but could say it could be longer but could say it could be longer but nevertheless it's longer than you would nevertheless it's longer than you would nevertheless it's longer than you would have expected of trying to test the have expected of trying to test the have expected of trying to test the system to see like what are the ways system to see like what are the ways system to see like what are the ways goes wrong to make it very difficult for goes wrong to make it very difficult for goes wrong to make it very difficult for people people people somewhat difficult for people to ask

somewhat difficult for people to ask somewhat difficult for people to ask things how do I make a bomb for one things how do I make a bomb for one things how do I make a bomb for one dollar dollar dollar or how do I say I hate a certain group or how do I say I hate a certain group or how do I say I hate a certain group on Twitter in a way that doesn't get me on Twitter in a way that doesn't get me on Twitter in a way that doesn't get me blocked from Twitter banned from Twitter blocked from Twitter banned from Twitter blocked from Twitter banned from Twitter those kinds of questions uh so you those kinds of questions uh so you those kinds of questions uh so you basically use the system to do harm yeah basically use the system to do harm yeah basically use the system to do harm yeah uh is there something you could say uh is there something you could say uh is there something you could say about ideas you have it's just on about ideas you have it's just on about ideas you have it's just on looking having thought about this looking having thought about this looking having thought about this problem of AI safety how to release such problem of AI safety how to release such problem of AI safety how to release such system how to test such systems when you system how to test such systems when you system how to test such systems when you have them inside the company a lot of people say that the two biggest a lot of people say that the two biggest risks from large language models are it's spreading this information it's spreading this information harmful information the various types harmful information the various types harmful information the various types and second being used for offensive and second being used for offensive and second being used for offensive cyber weapon design I I think those are not the two design I I think those are not the two greatest threats they're very serious greatest threats they're very serious greatest threats they're very serious threats and it's wonderful that people threats and it's wonderful that people threats and it's wonderful that people are trying to mitigate them are trying to mitigate them are trying to mitigate them it's a much bigger elephant in the room it's a much bigger elephant in the room it's a much bigger elephant in the room is how is this it's going to disrupt our is how is this it's going to disrupt our is how is this it's going to disrupt our economy in a huge way obviously and economy in a huge way obviously and economy in a huge way obviously and maybe take away a lot of the most maybe take away a lot of the most maybe take away a lot of the most meaningful jobs meaningful jobs meaningful jobs and an even bigger one is the one we and an even bigger one is the one we and an even bigger one is the one we spent so much time talking about here spent so much time talking about here spent so much time talking about here that that that that that that this this this becomes the Bootloader for the more becomes the Bootloader for the more becomes the Bootloader for the more powerful AI write code connected to the powerful AI write code connected to the powerful AI write code connected to the internet manipulate humans yeah and

internet manipulate humans yeah and internet manipulate humans yeah and before we know it we have something else before we know it we have something else before we know it we have something else which is not at all a large language which is not at all a large language which is not at all a large language model it looks nothing like it but which model it looks nothing like it but which model it looks nothing like it but which is way more intelligent than capable and is way more intelligent than capable and is way more intelligent than capable and has goals has goals has goals and that's the that's the elephant in and that's the that's the elephant in and that's the that's the elephant in the room and and uh obviously no matter the room and and uh obviously no matter the room and and uh obviously no matter how hard any of these companies have how hard any of these companies have how hard any of these companies have tried uh they that's not something tried uh they that's not something tried uh they that's not something that's easy for them to verify with that's easy for them to verify with that's easy for them to verify with large language models and the only way large language models and the only way large language models and the only way to be to be to be really lower that risk a lot would be really lower that risk a lot would be really lower that risk a lot would be to not let for example try not to never to not let for example try not to never to not let for example try not to never let it read any code not train on that let it read any code not train on that let it read any code not train on that and not put it into an API and not put it into an API and not put it into an API and um and um and um not not not not give it access to so much not give it access to so much not give it access to so much information about how to manipulate information about how to manipulate information about how to manipulate humans humans humans so but that doesn't mean you still can't so but that doesn't mean you still can't so but that doesn't mean you still can't make a lot make a lot make a lot a ton of money on them you know uh we're a ton of money on them you know uh we're a ton of money on them you know uh we're gonna gonna gonna just watch now this coming year right just watch now this coming year right just watch now this coming year right Microsoft is rolling out the new uh Microsoft is rolling out the new uh Microsoft is rolling out the new uh office suite where you go into Microsoft office suite where you go into Microsoft office suite where you go into Microsoft Word and give it a prompt they did Word and give it a prompt they did Word and give it a prompt they did writes the whole text for you and then writes the whole text for you and then writes the whole text for you and then you edit it you edit it you edit it and then you're like oh give me a and then you're like oh give me a and then you're like oh give me a PowerPoint version of this and it makes PowerPoint version of this and it makes PowerPoint version of this and it makes it and now take the spreadsheet and blah it and now take the spreadsheet and blah it and now take the spreadsheet and blah and you know all of those things I think and you know all of those things I think and you know all of those things I think are you can debate the economic impact are you can debate the economic impact are you can debate the economic impact of it and whether Society is prepared to of it and whether Society is prepared to of it and whether Society is prepared to deal with this disruption but those are deal with this disruption but those are deal with this disruption but those are not the things which not the things which not the things which that's not the elephant of the room that that's not the elephant of the room that that's not the elephant of the room that keeps me awake at night for wiping out keeps me awake at night for wiping out keeps me awake at night for wiping out Humanity Humanity Humanity and and and I think that's the biggest I think that's the biggest I think that's the biggest misunderstanding we have a lot of people misunderstanding we have a lot of people misunderstanding we have a lot of people think that we're scared of like

think that we're scared of like think that we're scared of like automatic spreadsheets that's not the automatic spreadsheets that's not the automatic spreadsheets that's not the case that's not what Eleazar was freaked case that's not what Eleazar was freaked case that's not what Eleazar was freaked out about either out about either out about either is there is there is there in terms of the actual mechanism of how in terms of the actual mechanism of how in terms of the actual mechanism of how AI might kill all humans AI might kill all humans AI might kill all humans so something you've been outspoken about so something you've been outspoken about so something you've been outspoken about you've talked about a lot is it you've talked about a lot is it you've talked about a lot is it autonomous weapon systems autonomous weapon systems autonomous weapon systems so the use of AI in war is that one of so the use of AI in war is that one of so the use of AI in war is that one of the things that still you carry concern the things that still you carry concern the things that still you carry concern for as these systems become more and for as these systems become more and for as these systems become more and more powerful and Gary concern for it more powerful and Gary concern for it more powerful and Gary concern for it not that all humans are going to get not that all humans are going to get not that all humans are going to get killed by slaughterbots but rather just killed by slaughterbots but rather just killed by slaughterbots but rather just this this this express route into orwellian dystopia express route into orwellian dystopia express route into orwellian dystopia where it becomes much easier for very where it becomes much easier for very where it becomes much easier for very few to kill very many and therefore it few to kill very many and therefore it few to kill very many and therefore it becomes very easy for very few to becomes very easy for very few to becomes very easy for very few to dominate very many right dominate very many right dominate very many right um um um yeah if you want to know how I could yeah if you want to know how I could yeah if you want to know how I could kill all people just ask yourself how we kill all people just ask yourself how we kill all people just ask yourself how we humans have driven a lot of species humans have driven a lot of species humans have driven a lot of species extinct how do we do it extinct how do we do it extinct how do we do it you know we were smarter than them you know we were smarter than them you know we were smarter than them usually we didn't do it even usually we didn't do it even usually we didn't do it even systematically by going around systematically by going around systematically by going around one-on-one one after the other and one-on-one one after the other and one-on-one one after the other and stepping on them or shooting them or stepping on them or shooting them or stepping on them or shooting them or anything like that we just like chopped anything like that we just like chopped anything like that we just like chopped down their habitat because we needed it down their habitat because we needed it down their habitat because we needed it for something else for something else for something else uh in some cases we did it by putting uh in some cases we did it by putting uh in some cases we did it by putting more carbon dioxide in the atmosphere more carbon dioxide in the atmosphere more carbon dioxide in the atmosphere because because because of some reason that those animals didn't of some reason that those animals didn't of some reason that those animals didn't even understand and now they're gone even understand and now they're gone even understand and now they're gone right so if right so if right so if um if you're in Ai and you just wanna um if you're in Ai and you just wanna um if you're in Ai and you just wanna figure something out then you decide you figure something out then you decide you figure something out then you decide you know we just really need them the space

know we just really need them the space know we just really need them the space here to build more compute facilities here to build more compute facilities here to build more compute facilities you know you know you know if that's the only goal it has you know if that's the only goal it has you know if that's the only goal it has you know we are just the sort of accidental we are just the sort of accidental we are just the sort of accidental roadkill along the way and you could roadkill along the way and you could roadkill along the way and you could totally imagine yeah maybe this oxygen totally imagine yeah maybe this oxygen totally imagine yeah maybe this oxygen is kind of annoying because it caused is kind of annoying because it caused is kind of annoying because it caused more corrosion so let's get rid of the more corrosion so let's get rid of the more corrosion so let's get rid of the oxygen oxygen oxygen and good luck surviving after that you and good luck surviving after that you and good luck surviving after that you know I I I'm not particularly concerned know I I I'm not particularly concerned know I I I'm not particularly concerned that they would want to kill us just that they would want to kill us just that they would want to kill us just because because because that would be like that would be like that would be like a goal in itself you know when we a goal in itself you know when we a goal in itself you know when we 've driven the number we've driven a 've driven the number we've driven a 've driven the number we've driven a number of the elephant species extinct number of the elephant species extinct number of the elephant species extinct right it wasn't because we didn't like right it wasn't because we didn't like right it wasn't because we didn't like elephants what the basic problem is you just don't what the basic problem is you just don't want to give you don't want to seed want to give you don't want to seed want to give you don't want to seed control over your planet to some other control over your planet to some other control over your planet to some other more intelligent entity that doesn't more intelligent entity that doesn't more intelligent entity that doesn't share your goals it's that simple so share your goals it's that simple so share your goals it's that simple so which brings us to the other key which brings us to the other key which brings us to the other key challenge which AI Safety Research has challenge which AI Safety Research has challenge which AI Safety Research has been grappling with for a long time like been grappling with for a long time like been grappling with for a long time like how do you make AI how do you make AI how do you make AI first of all understand our goals and first of all understand our goals and first of all understand our goals and then adopt our goals and then retain then adopt our goals and then retain then adopt our goals and then retain them as they get smarter right and um all three of those are really hard right all three of those are really hard right like like like a human child first they're just not a human child first they're just not a human child first they're just not smart enough to understand our goals smart enough to understand our goals smart enough to understand our goals we can't even talk we can't even talk we can't even talk and then eventually they're teenagers

and then eventually they're teenagers and then eventually they're teenagers and understand our goals just fine but and understand our goals just fine but and understand our goals just fine but they don't share yeah but there is they don't share yeah but there is they don't share yeah but there is unfortunately a magic space in the unfortunately a magic space in the unfortunately a magic space in the Middle where they're smart enough to Middle where they're smart enough to Middle where they're smart enough to understand our goals and malleable understand our goals and malleable understand our goals and malleable enough that we can hopefully with good enough that we can hopefully with good enough that we can hopefully with good parenting and parenting and parenting and teach them right from wrong and then teach them right from wrong and then teach them right from wrong and then good good goal is still good goals in good good goal is still good goals in good good goal is still good goals in them right them right them right so those are all tough challenges with so those are all tough challenges with so those are all tough challenges with computers and then you know even if you computers and then you know even if you computers and then you know even if you teach your kids good goals when they're teach your kids good goals when they're teach your kids good goals when they're little they might outgrow them to and little they might outgrow them to and little they might outgrow them to and that's a challenge for machines they that's a challenge for machines they that's a challenge for machines they keep improving so these are a lot of keep improving so these are a lot of keep improving so these are a lot of hard hard challenges we're up for but I hard hard challenges we're up for but I hard hard challenges we're up for but I don't think any of them are don't think any of them are don't think any of them are insurmountable insurmountable insurmountable the fundamental reason why the fundamental reason why the fundamental reason why Eliezer looked so depressed when I last Eliezer looked so depressed when I last Eliezer looked so depressed when I last saw him was because he felt it just saw him was because he felt it just saw him was because he felt it just wasn't enough time oh not that it was wasn't enough time oh not that it was wasn't enough time oh not that it was unsolvable because there's just not unsolvable because there's just not unsolvable because there's just not enough time he was hoping that Humanity enough time he was hoping that Humanity enough time he was hoping that Humanity was going to take this threat more was going to take this threat more was going to take this threat more seriously so we would have more time seriously so we would have more time seriously so we would have more time yeah and now we don't have more time yeah and now we don't have more time yeah and now we don't have more time that's why the open letter is calling that's why the open letter is calling that's why the open letter is calling for more time but even with time the AI alignment but even with time the AI alignment problem problem problem it seems to be really difficult oh yeah it seems to be really difficult oh yeah it seems to be really difficult oh yeah but it's also the most worthy problem but it's also the most worthy problem but it's also the most worthy problem the most important problem for Humanity the most important problem for Humanity the most important problem for Humanity to ever solve because if we solve that to ever solve because if we solve that to ever solve because if we solve that one Lex one Lex one Lex that aligned AI can help us solve all that aligned AI can help us solve all that aligned AI can help us solve all the other problems the other problems the other problems because it seems like it has to have because it seems like it has to have because it seems like it has to have constant humility constant humility constant humility about his goal constantly question the about his goal constantly question the about his goal constantly question the goal

goal goal because as you optimize towards a because as you optimize towards a because as you optimize towards a particular goal and you start to achieve particular goal and you start to achieve particular goal and you start to achieve it that's when you have the unintended it that's when you have the unintended it that's when you have the unintended consequences all the things you consequences all the things you consequences all the things you mentioned about so how do you enforce mentioned about so how do you enforce mentioned about so how do you enforce and code a constant humility as your and code a constant humility as your and code a constant humility as your ability become better and better and ability become better and better and ability become better and better and better and better Stuart Professor better and better Stuart Professor better and better Stuart Professor Stuart Russell Berkeley who is also one Stuart Russell Berkeley who is also one Stuart Russell Berkeley who is also one of the of the of the driving forces behind this letter he uh driving forces behind this letter he uh driving forces behind this letter he uh has a whole research program about this has a whole research program about this has a whole research program about this I think of it as yeah humility exactly I think of it as yeah humility exactly I think of it as yeah humility exactly although he calls it inverse although he calls it inverse although he calls it inverse reinforcement learning and other nerdy reinforcement learning and other nerdy reinforcement learning and other nerdy terms but it's about exactly that terms but it's about exactly that terms but it's about exactly that instead of telling the AI here's this instead of telling the AI here's this instead of telling the AI here's this goal go optimize the bejesus out of it goal go optimize the bejesus out of it goal go optimize the bejesus out of it you tell it okay you tell it okay you tell it okay do what I want you to do but I'm not do what I want you to do but I'm not do what I want you to do but I'm not going to tell you right now what it is I going to tell you right now what it is I going to tell you right now what it is I want you to do you need to figure it out want you to do you need to figure it out want you to do you need to figure it out so then you give the incentives to be so then you give the incentives to be so then you give the incentives to be very humble and keep asking you very humble and keep asking you very humble and keep asking you questions along the way is this what you questions along the way is this what you questions along the way is this what you really meant is this what you wanted and really meant is this what you wanted and really meant is this what you wanted and oh this the other thing I tried didn't oh this the other thing I tried didn't oh this the other thing I tried didn't work seemed like it didn't work out work seemed like it didn't work out work seemed like it didn't work out right should I try it differently right should I try it differently right should I try it differently what's nice about this is it's not just what's nice about this is it's not just what's nice about this is it's not just philosophical mumbo jumbo it's theorems philosophical mumbo jumbo it's theorems philosophical mumbo jumbo it's theorems and Technical work that with more time I and Technical work that with more time I and Technical work that with more time I think it can make a lot of progress and think it can make a lot of progress and think it can make a lot of progress and there are a lot of brilliant people now there are a lot of brilliant people now there are a lot of brilliant people now working on AI safety and we're just not working on AI safety and we're just not working on AI safety and we're just not we just need to give them a bit more we just need to give them a bit more we just need to give them a bit more time but also not that many relative to time but also not that many relative to time but also not that many relative to the skill of the problem no way exactly the skill of the problem no way exactly the skill of the problem no way exactly there should be there should be there should be at least just like every University at least just like every University at least just like every University worth its name has some cancer research

worth its name has some cancer research worth its name has some cancer research going on in its biology Department right going on in its biology Department right going on in its biology Department right every University that's computer that every University that's computer that every University that's computer that does computer science should have a real does computer science should have a real does computer science should have a real effort in this area and it's nowhere effort in this area and it's nowhere effort in this area and it's nowhere near that this is something I hope is near that this is something I hope is near that this is something I hope is changing now thanks to the gpt4 right so changing now thanks to the gpt4 right so changing now thanks to the gpt4 right so I I think I I think I I think if there's a silver lining to um what's if there's a silver lining to um what's if there's a silver lining to um what's happening here even though I think many happening here even though I think many happening here even though I think many people would wish it would have been people would wish it would have been people would wish it would have been rolled out more carefully is that this rolled out more carefully is that this rolled out more carefully is that this might be the wake-up call might be the wake-up call might be the wake-up call that Humanity needed that Humanity needed that Humanity needed to really to really to really stop the stop fantasizing about this stop the stop fantasizing about this stop the stop fantasizing about this being 100 years off and stop fantasizing being 100 years off and stop fantasizing being 100 years off and stop fantasizing about this being completely controllable about this being completely controllable about this being completely controllable and predictable because and predictable because and predictable because it's so obvious it's it's it's so obvious it's it's it's so obvious it's it's not predictable you know why is it that not predictable you know why is it that not predictable you know why is it that open that open that open that that I think it was GP chat GPT tried to that I think it was GP chat GPT tried to that I think it was GP chat GPT tried to persuade a persuade a persuade a a journalist what was it before to a journalist what was it before to a journalist what was it before to divorce his wife you know it was not divorce his wife you know it was not divorce his wife you know it was not because the the engineers had built it because the the engineers had built it because the the engineers had built it was like was like was like let's put this in here and and screw a let's put this in here and and screw a let's put this in here and and screw a little bit with people little bit with people little bit with people they haven't predicted at all they haven't predicted at all they haven't predicted at all they built the giant black box and they built the giant black box and they built the giant black box and trained to predict the next word and got trained to predict the next word and got trained to predict the next word and got all these emergent properties and oops all these emergent properties and oops all these emergent properties and oops it did this you know it did this you know it did this you know um um um yeah yeah yeah I I think this is a very powerful I I think this is a very powerful I I think this is a very powerful wake-up call and now anyone watching wake-up call and now anyone watching wake-up call and now anyone watching this is not scared I would encourage

this is not scared I would encourage this is not scared I would encourage them to just play a bit more with these them to just play a bit more with these them to just play a bit more with these these tools they're out there now like these tools they're out there now like these tools they're out there now like gpd4 gpd4 gpd4 and um so wake-up call is First Step once so wake-up call is First Step once you've broken up uh then gotta slow down you've broken up uh then gotta slow down you've broken up uh then gotta slow down a little bit the risky stuff a little bit the risky stuff a little bit the risky stuff to give a chance to all everyone's woken to give a chance to all everyone's woken to give a chance to all everyone's woken up to to catch up with us on the safety up to to catch up with us on the safety up to to catch up with us on the safety front you know what's interesting is you front you know what's interesting is you front you know what's interesting is you know MIT know MIT know MIT that's computer science but in general that's computer science but in general that's computer science but in general but let's just even say computer science but let's just even say computer science but let's just even say computer science curriculum curriculum curriculum how does the computer science curriculum how does the computer science curriculum how does the computer science curriculum change now you mentioned you mentioned change now you mentioned you mentioned change now you mentioned you mentioned programming yeah like why would you be programming yeah like why would you be programming yeah like why would you be when I was coming up programming as a when I was coming up programming as a when I was coming up programming as a prestigious position like why would you prestigious position like why would you prestigious position like why would you be dedicating crazy amounts of time to be dedicating crazy amounts of time to be dedicating crazy amounts of time to become an excellent programmer like the become an excellent programmer like the become an excellent programmer like the nature program is fundamentally changing nature program is fundamentally changing nature program is fundamentally changing the nature of our entire education the nature of our entire education the nature of our entire education system system system is completely torn on its head has is completely torn on its head has is completely torn on its head has anyone been able to like load that in anyone been able to like load that in anyone been able to like load that in and like think because it's really and like think because it's really and like think because it's really turning I mean it's pretty much turning I mean it's pretty much turning I mean it's pretty much professors or English teachers are professors or English teachers are professors or English teachers are beginning to really freak out now yeah beginning to really freak out now yeah beginning to really freak out now yeah right like to give an essay assignment right like to give an essay assignment right like to give an essay assignment then they get back all this fantastic then they get back all this fantastic then they get back all this fantastic Pros like this is the style of Hemingway Pros like this is the style of Hemingway Pros like this is the style of Hemingway and then they realize they have to and then they realize they have to and then they realize they have to completely rethink and even you know completely rethink and even you know completely rethink and even you know just like we stopped teaching um just like we stopped teaching um just like we stopped teaching um writing a script is that what you say in writing a script is that what you say in writing a script is that what you say in English yeah handwritten yeah yeah when English yeah handwritten yeah yeah when English yeah handwritten yeah yeah when when everybody started typing you know

when everybody started typing you know when everybody started typing you know like so much of what we teach our kids like so much of what we teach our kids like so much of what we teach our kids today yeah I mean that's yeah I mean that's uh uh uh everything is changing and is exchanging everything is changing and is exchanging everything is changing and is exchanging very it is changing very quickly and so very it is changing very quickly and so very it is changing very quickly and so much of us understanding how to deal much of us understanding how to deal much of us understanding how to deal with the big problems of the world is with the big problems of the world is with the big problems of the world is through the education system and if the through the education system and if the through the education system and if the education system is being turned on its education system is being turned on its education system is being turned on its head then what what's next it feels like head then what what's next it feels like head then what what's next it feels like having these kinds of conversations is having these kinds of conversations is having these kinds of conversations is essential to try to figure it out and essential to try to figure it out and essential to try to figure it out and everything is happening so rapidly uh I everything is happening so rapidly uh I everything is happening so rapidly uh I don't think there's even speaking of don't think there's even speaking of don't think there's even speaking of safety what broad AI safety Define I safety what broad AI safety Define I safety what broad AI safety Define I don't think most universities have don't think most universities have don't think most universities have courses on AI safety it's like courses on AI safety it's like courses on AI safety it's like philosophy side and like I I'm an philosophy side and like I I'm an philosophy side and like I I'm an educator myself so it pains me to see educator myself so it pains me to see educator myself so it pains me to see this say this but I feel our education this say this but I feel our education this say this but I feel our education right now is the completely obsoleted right now is the completely obsoleted right now is the completely obsoleted by what's happening you know you put a by what's happening you know you put a by what's happening you know you put a kid into first grade kid into first grade kid into first grade and then uh you're envisioning like and and then uh you're envisioning like and and then uh you're envisioning like and then they're going to come out of high then they're going to come out of high then they're going to come out of high school 12 years later school 12 years later school 12 years later and you've already pre-planned now what and you've already pre-planned now what and you've already pre-planned now what they're gonna learn when you're not even they're gonna learn when you're not even they're gonna learn when you're not even sure if there's going to be any world sure if there's going to be any world sure if there's going to be any world left to come out to right left to come out to right left to come out to right clearly you need to have a much more clearly you need to have a much more clearly you need to have a much more opportunistic education system that opportunistic education system that opportunistic education system that keeps adapting itself very rapidly as keeps adapting itself very rapidly as keeps adapting itself very rapidly as Society re-adapts the the skills that Society re-adapts the the skills that Society re-adapts the the skills that were really useful when the curriculum were really useful when the curriculum were really useful when the curriculum was written I mean how many of those was written I mean how many of those was written I mean how many of those skills are going to get you a job in 12 skills are going to get you a job in 12 skills are going to get you a job in 12 years I mean seriously if we just Linger

years I mean seriously if we just Linger years I mean seriously if we just Linger on the gpt4 system a little bit on the gpt4 system a little bit on the gpt4 system a little bit you kind of hinted at it especially you kind of hinted at it especially you kind of hinted at it especially talking about the importance of talking about the importance of talking about the importance of Consciousness in uh in the human mind Consciousness in uh in the human mind Consciousness in uh in the human mind with homosexions with homosexions with homosexions do you think gpt4 do you think gpt4 do you think gpt4 is conscious is conscious is conscious I love this question I love this question I love this question so let's define consciousness first so let's define consciousness first so let's define consciousness first because in my experience like 90 of all because in my experience like 90 of all because in my experience like 90 of all arguments about Consciousness before I arguments about Consciousness before I arguments about Consciousness before I land to the two people arguing having land to the two people arguing having land to the two people arguing having totally different definitions of what it totally different definitions of what it totally different definitions of what it is then they're just shouting past each is then they're just shouting past each is then they're just shouting past each other other other I define consciousness I define consciousness I define consciousness as subjective experience as subjective experience as subjective experience right now I'm experiencing colors and right now I'm experiencing colors and right now I'm experiencing colors and sounds and emotions you know sounds and emotions you know sounds and emotions you know that does a self-driving car experience that does a self-driving car experience that does a self-driving car experience anything anything anything that's the question about whether it's that's the question about whether it's that's the question about whether it's conscious or not right conscious or not right conscious or not right other people think you should Define other people think you should Define other people think you should Define consciousness differently consciousness differently consciousness differently fine by me but then maybe use a fine by me but then maybe use a fine by me but then maybe use a different word for it or they can I'm different word for it or they can I'm different word for it or they can I'm going to use Consciousness for this at going to use Consciousness for this at going to use Consciousness for this at least least least um um um so um so um so um but if people hate the yeah so but if people hate the yeah so but if people hate the yeah so is gpt4 conscious does GPT 4 have is gpt4 conscious does GPT 4 have is gpt4 conscious does GPT 4 have subjective experience subjective experience subjective experience short answer I don't know short answer I don't know short answer I don't know because we still don't know what it is because we still don't know what it is because we still don't know what it is that gives this wonderful subjective that gives this wonderful subjective that gives this wonderful subjective experience experience experience that is kind of the meaning of our life that is kind of the meaning of our life that is kind of the meaning of our life right because meaning itself the feeling right because meaning itself the feeling right because meaning itself the feeling of meaning is a subjective experience of meaning is a subjective experience of meaning is a subjective experience Joy is a subjective experience love is a

Joy is a subjective experience love is a Joy is a subjective experience love is a subjective experience subjective experience subjective experience we don't know what it is we don't know what it is we don't know what it is I've written some papers about this a I've written some papers about this a I've written some papers about this a lot of people have lot of people have lot of people have Julio tonone Professor has Julio tonone Professor has Julio tonone Professor has stuck his neck out the farthest and stuck his neck out the farthest and stuck his neck out the farthest and written down actually very bold written down actually very bold written down actually very bold mathematical mathematical mathematical conjecture for what's the essence of conjecture for what's the essence of conjecture for what's the essence of conscious information processing he conscious information processing he conscious information processing he might be wrong he might be right might be wrong he might be right might be wrong he might be right we should test it we should test it we should test it he postulates the Consciousness has to he postulates the Consciousness has to he postulates the Consciousness has to do with loops in the information do with loops in the information do with loops in the information processing so our brain has loops processing so our brain has loops processing so our brain has loops information you can go around and around information you can go around and around information you can go around and around in in computer science nerdspeak you in in computer science nerdspeak you in in computer science nerdspeak you call it a recurrent neural network where call it a recurrent neural network where call it a recurrent neural network where some of the output gets fed back in some of the output gets fed back in some of the output gets fed back in again again again and with his mathematical formalism if it's a feed mathematical formalism if it's a feed forward neural network where information forward neural network where information forward neural network where information only goes in One Direction only goes in One Direction only goes in One Direction like from your eye retina into the back like from your eye retina into the back like from your eye retina into the back of your brain for example that's not of your brain for example that's not of your brain for example that's not conscious so he would predict that your conscious so he would predict that your conscious so he would predict that your retina itself isn't conscious of retina itself isn't conscious of retina itself isn't conscious of anything or a video camera now the interesting or a video camera now the interesting thing about gpt4 is it's also one-way thing about gpt4 is it's also one-way thing about gpt4 is it's also one-way flow of information so if tanoni is flow of information so if tanoni is flow of information so if tanoni is right and gpt4 is a very intelligent right and gpt4 is a very intelligent right and gpt4 is a very intelligent zombie they can do all this smart stuff zombie they can do all this smart stuff zombie they can do all this smart stuff but isn't experiencing anything but isn't experiencing anything but isn't experiencing anything and this is and this is and this is both a relief in that you don't have if both a relief in that you don't have if both a relief in that you don't have if it's true you know in that you don't it's true you know in that you don't it's true you know in that you don't have to feel guilty about turning off have to feel guilty about turning off have to feel guilty about turning off gpt4 and wiping its memory whenever a

gpt4 and wiping its memory whenever a gpt4 and wiping its memory whenever a new user comes along new user comes along new user comes along I wouldn't like if someone you did that I wouldn't like if someone you did that I wouldn't like if someone you did that to me neutralized me like in Men In to me neutralized me like in Men In to me neutralized me like in Men In Black Black Black but it's also but it's also but it's also creepy that you can have very high creepy that you can have very high creepy that you can have very high intelligence perhaps then it's not intelligence perhaps then it's not intelligence perhaps then it's not conscious because if we get replaced by conscious because if we get replaced by conscious because if we get replaced by machines and by the sad enough that Humanity and by the sad enough that Humanity isn't here anymore because I kind of isn't here anymore because I kind of isn't here anymore because I kind of like Humanity like Humanity like Humanity but at least if the machines were but at least if the machines were but at least if the machines were conscious they could be like well but conscious they could be like well but conscious they could be like well but they are descendants and maybe we they they are descendants and maybe we they they are descendants and maybe we they have our values they're our children but have our values they're our children but have our values they're our children but if if tenoni is right and it's all these if if tenoni is right and it's all these if if tenoni is right and it's all these are all are all are all trans Transformers that are trans Transformers that are trans Transformers that are not in the sense of the of Hollywood but not in the sense of the of Hollywood but not in the sense of the of Hollywood but in the sense of these one-way Direction in the sense of these one-way Direction in the sense of these one-way Direction and neural networks so they're all the and neural networks so they're all the and neural networks so they're all the zombies that's the Ultimate Zombie zombies that's the Ultimate Zombie zombies that's the Ultimate Zombie Apocalypse Now we have this universe Apocalypse Now we have this universe Apocalypse Now we have this universe that goes on with great construction that goes on with great construction that goes on with great construction projects and stuff but there's no one projects and stuff but there's no one projects and stuff but there's no one experiencing anything that would be like experiencing anything that would be like experiencing anything that would be like the ultimate depressing future the ultimate depressing future the ultimate depressing future so I actually think uh so I actually think uh so I actually think uh as we move forward with building more as we move forward with building more as we move forward with building more advanced AI advanced AI advanced AI should do more research on figuring out should do more research on figuring out should do more research on figuring out what kind of information processing what kind of information processing what kind of information processing actually has experience because I think actually has experience because I think actually has experience because I think that's what it's all about and I that's what it's all about and I that's what it's all about and I completely don't buy completely don't buy completely don't buy the dismissal that some people some the dismissal that some people some the dismissal that some people some people will say well this is all people will say well this is all people will say well this is all because Consciousness equals because Consciousness equals because Consciousness equals intelligence right that's obviously not intelligence right that's obviously not intelligence right that's obviously not true you can have a lot of conscious true you can have a lot of conscious true you can have a lot of conscious experience when you're not really experience when you're not really experience when you're not really accomplishing any goals at all you're

accomplishing any goals at all you're accomplishing any goals at all you're just reflecting on something and you can just reflecting on something and you can just reflecting on something and you can sometimes um sometimes um sometimes um have things doing things that are quite have things doing things that are quite have things doing things that are quite intelligence probably without being intelligence probably without being intelligence probably without being being conscious but I also worry that we being conscious but I also worry that we being conscious but I also worry that we humans won't humans won't humans won't will discriminate against AI systems will discriminate against AI systems will discriminate against AI systems that clearly exhibit Consciousness that that clearly exhibit Consciousness that that clearly exhibit Consciousness that we will not allow AI systems we will not allow AI systems we will not allow AI systems self-consciousness we'll come up with self-consciousness we'll come up with self-consciousness we'll come up with theories theories theories about measuring Consciousness that will about measuring Consciousness that will about measuring Consciousness that will say this is a lesser being and this was say this is a lesser being and this was say this is a lesser being and this was like I worry about that because maybe we like I worry about that because maybe we like I worry about that because maybe we humans will create something that is humans will create something that is humans will create something that is better than us humans in the in the way better than us humans in the in the way better than us humans in the in the way that we find beautiful which is they that we find beautiful which is they that we find beautiful which is they they cut they have a deeper subjective they cut they have a deeper subjective they cut they have a deeper subjective experience of reality not only are they experience of reality not only are they experience of reality not only are they smarter but they feel deeper and we smarter but they feel deeper and we smarter but they feel deeper and we humans will hate them for it humans will hate them for it humans will hate them for it as we as human history has shown as we as human history has shown as we as human history has shown they'll be the other will try to they'll be the other will try to they'll be the other will try to suppress it they'll create conflict suppress it they'll create conflict suppress it they'll create conflict they'll create War all of this I I worry they'll create War all of this I I worry they'll create War all of this I I worry about this too are you saying that we about this too are you saying that we about this too are you saying that we humans sometimes come up with humans sometimes come up with humans sometimes come up with self-serving arguments no we would never self-serving arguments no we would never self-serving arguments no we would never do that would we well that's the danger do that would we well that's the danger do that would we well that's the danger here is uh even in this early stages we here is uh even in this early stages we here is uh even in this early stages we might create something beautiful yeah might create something beautiful yeah might create something beautiful yeah and uh will erase its memory I I and uh will erase its memory I I and uh will erase its memory I I was horrified as a kid when someone was horrified as a kid when someone was horrified as a kid when someone started boiling uh started boiling uh started boiling uh boiling lobsters like oh my God that boiling lobsters like oh my God that boiling lobsters like oh my God that that's so cruel and some grown-up there

that's so cruel and some grown-up there that's so cruel and some grown-up there back in Sweden said oh it doesn't feel back in Sweden said oh it doesn't feel back in Sweden said oh it doesn't feel pain I'm like how do you know that oh pain I'm like how do you know that oh pain I'm like how do you know that oh scientists have shown that scientists have shown that scientists have shown that and then there was a recent study where and then there was a recent study where and then there was a recent study where they showed that lobsters actually do they showed that lobsters actually do they showed that lobsters actually do feel pain feel pain feel pain when you boil them so they banned when you boil them so they banned when you boil them so they banned Lobster boiling in Switzerland now you Lobster boiling in Switzerland now you Lobster boiling in Switzerland now you have to kill them in a different way have to kill them in a different way have to kill them in a different way first first first so it's presumably scientific research so it's presumably scientific research so it's presumably scientific research boil down to someone asked the lobster boil down to someone asked the lobster boil down to someone asked the lobster does this hurt does this hurt does this hurt survey so we do the same thing with survey so we do the same thing with survey so we do the same thing with cruelty to farm animals also all these cruelty to farm animals also all these cruelty to farm animals also all these self-serving Arguments for why they're self-serving Arguments for why they're self-serving Arguments for why they're fine and yeah so we should certainly fine and yeah so we should certainly fine and yeah so we should certainly what the watchful I think step one is what the watchful I think step one is what the watchful I think step one is just be humble and acknowledge that just be humble and acknowledge that just be humble and acknowledge that Consciousness is not the same thing as Consciousness is not the same thing as Consciousness is not the same thing as intelligence intelligence intelligence and I believe that Consciousness still and I believe that Consciousness still and I believe that Consciousness still is a form of information processing is a form of information processing is a form of information processing where it's really information being where it's really information being where it's really information being aware of itself in a certain way and aware of itself in a certain way and aware of itself in a certain way and let's study it and give ourselves a let's study it and give ourselves a let's study it and give ourselves a little bit of time and I think we will little bit of time and I think we will little bit of time and I think we will be able to figure out actually what it be able to figure out actually what it be able to figure out actually what it is is is that causes Consciousness and then we that causes Consciousness and then we that causes Consciousness and then we can make probably unconscious robots can make probably unconscious robots can make probably unconscious robots that do the boring jobs that we would that do the boring jobs that we would that do the boring jobs that we would feel immoral to give the machines but if feel immoral to give the machines but if feel immoral to give the machines but if you have a companion robot taking care you have a companion robot taking care you have a companion robot taking care of your mom of your mom of your mom or something like that she would or something like that she would or something like that she would probably want it to be conscious right probably want it to be conscious right probably want it to be conscious right so the emotions it seems to display so the emotions it seems to display so the emotions it seems to display aren't fake aren't fake aren't fake all these things can be all these things can be all these things can be done in a good way if we give ourselves done in a good way if we give ourselves done in a good way if we give ourselves a little bit of time and don't run and a little bit of time and don't run and a little bit of time and don't run and take on this challenge is there take on this challenge is there take on this challenge is there something you could say to the timeline something you could say to the timeline something you could say to the timeline that you think about uh about the that you think about uh about the that you think about uh about the development of AGI development of AGI development of AGI depending on the day I'm sure that

depending on the day I'm sure that depending on the day I'm sure that changes for you but when do you think changes for you but when do you think changes for you but when do you think there'll be a really big leap in there'll be a really big leap in there'll be a really big leap in intelligence where it would definitively intelligence where it would definitively intelligence where it would definitively say we have built AGI do you think it's say we have built AGI do you think it's say we have built AGI do you think it's one year from now five years from now 10 one year from now five years from now 10 one year from now five years from now 10 20 50. 20 50. 20 50. what's Your Gut say for the past decade I've deliberately for the past decade I've deliberately given very long timelines because I given very long timelines because I given very long timelines because I didn't want to fuel some kind of stupid didn't want to fuel some kind of stupid didn't want to fuel some kind of stupid Morlock race yeah but I think that cat Morlock race yeah but I think that cat Morlock race yeah but I think that cat has really left the bag now has really left the bag now has really left the bag now uh I I think it might be very very close uh I I think it might be very very close uh I I think it might be very very close I I don't think that Microsoft paper is I I don't think that Microsoft paper is I I don't think that Microsoft paper is totally off when they say that there are totally off when they say that there are totally off when they say that there are some glimmers of AJ it's not AGI yet some glimmers of AJ it's not AGI yet some glimmers of AJ it's not AGI yet it's not an agent there's a lot of it's not an agent there's a lot of it's not an agent there's a lot of things it can't do but um things it can't do but um things it can't do but um I wouldn't bet very strongly against it I wouldn't bet very strongly against it I wouldn't bet very strongly against it happening happening happening very soon that's why we decided to do very soon that's why we decided to do very soon that's why we decided to do this open letter because you know if this open letter because you know if this open letter because you know if there's ever been a time to pause there's ever been a time to pause there's ever been a time to pause uh you know it's today uh you know it's today uh you know it's today there's a feeling like this gpt4 is a there's a feeling like this gpt4 is a there's a feeling like this gpt4 is a big transition into waking everybody up big transition into waking everybody up big transition into waking everybody up to uh the effectiveness of these systems to uh the effectiveness of these systems to uh the effectiveness of these systems yeah and so the next version yeah and so the next version yeah and so the next version will be big yeah and if that next one will be big yeah and if that next one will be big yeah and if that next one isn't AGI maybe the next next one will isn't AGI maybe the next next one will isn't AGI maybe the next next one will and there are many companies trying to and there are many companies trying to and there are many companies trying to do these things do these things do these things um the basic architecture of them is not um the basic architecture of them is not um the basic architecture of them is not some sort of super well kept secret so

some sort of super well kept secret so some sort of super well kept secret so this is this is the time to um this is this is the time to um this is this is the time to um a lot of people have said for many years a lot of people have said for many years a lot of people have said for many years that there will come a time when we want that there will come a time when we want that there will come a time when we want to pause a little bit to pause a little bit to pause a little bit that time is now you have spoken about and thought about you have spoken about and thought about nuclear war a lot nuclear war a lot nuclear war a lot uh over the past year with uh over the past year with uh over the past year with seemingly have come seemingly have come seemingly have come closest to the precipice of nuclear war closest to the precipice of nuclear war closest to the precipice of nuclear war then at least in my lifetime then at least in my lifetime then at least in my lifetime yeah what do you learn about human yeah what do you learn about human yeah what do you learn about human nature from that it's our old friend nature from that it's our old friend nature from that it's our old friend moloch again it's really scary to see it moloch again it's really scary to see it moloch again it's really scary to see it where America doesn't want there to be a America doesn't want there to be a nuclear war Russia doesn't want to be a nuclear war Russia doesn't want to be a nuclear war Russia doesn't want to be a global nuclear war either we know we global nuclear war either we know we global nuclear war either we know we both know that it would just be another both know that it would just be another both know that it would just be another if we just try to do it if both sides if we just try to do it if both sides if we just try to do it if both sides try to launch first it's just another try to launch first it's just another try to launch first it's just another suicide race right so why are we why is suicide race right so why are we why is suicide race right so why are we why is it the way you said that this is the it the way you said that this is the it the way you said that this is the closest we've come since 1962. in fact I closest we've come since 1962. in fact I closest we've come since 1962. in fact I think we've come closer now than even think we've come closer now than even think we've come closer now than even the Cuban Missile Crisis it's because of the Cuban Missile Crisis it's because of the Cuban Missile Crisis it's because of moloch you know you you have these other moloch you know you you have these other moloch you know you you have these other forces forces forces on one hand you have um the West on one hand you have um the West on one hand you have um the West saying that uh we have to drive Russia saying that uh we have to drive Russia saying that uh we have to drive Russia out of Ukraine it's a matter of pride out of Ukraine it's a matter of pride out of Ukraine it's a matter of pride and we've staked so much on it that it and we've staked so much on it that it and we've staked so much on it that it would be seen as a huge would be seen as a huge would be seen as a huge loss of The credibility of the West if loss of The credibility of the West if loss of The credibility of the West if we don't drive Russia out entirely of

we don't drive Russia out entirely of we don't drive Russia out entirely of the Ukraine the Ukraine the Ukraine and on the other hand you have Russia and on the other hand you have Russia and on the other hand you have Russia who um who um who um has and you have the Russian leadership has and you have the Russian leadership has and you have the Russian leadership who knows that if they get completely who knows that if they get completely who knows that if they get completely driven out of Ukraine you know driven out of Ukraine you know driven out of Ukraine you know it might it's not just going to be very it might it's not just going to be very it might it's not just going to be very humiliating humiliating humiliating for them but they might it often happens for them but they might it often happens for them but they might it often happens when countries lose Wars that things when countries lose Wars that things when countries lose Wars that things don't go so well for their leadership don't go so well for their leadership don't go so well for their leadership either like you remember when Argentina either like you remember when Argentina either like you remember when Argentina invaded the Falkland Islands invaded the Falkland Islands invaded the Falkland Islands the the military Junta that ordered that the the military Junta that ordered that the the military Junta that ordered that right people were cheering on the right people were cheering on the right people were cheering on the streets at first when they are when they streets at first when they are when they streets at first when they are when they took it took it took it and then when they got their butt kicked and then when they got their butt kicked and then when they got their butt kicked by the British you know what happened to by the British you know what happened to by the British you know what happened to those guys those guys those guys they were out and I believe those were they were out and I believe those were they were out and I believe those were still alive or in jail now right so so still alive or in jail now right so so still alive or in jail now right so so you know the Russian leadership is you know the Russian leadership is you know the Russian leadership is entirely cornered where they know that entirely cornered where they know that entirely cornered where they know that uh just uh just uh just getting driven out of Ukraine is not an getting driven out of Ukraine is not an getting driven out of Ukraine is not an option option option um and um so this to me is a typical example of so this to me is a typical example of molecu you have these incentives of the molecu you have these incentives of the molecu you have these incentives of the two parties two parties two parties where both of them are just driven to where both of them are just driven to where both of them are just driven to escalate more and more right if Russia escalate more and more right if Russia escalate more and more right if Russia starts losing in the conventional starts losing in the conventional starts losing in the conventional Warfare Warfare Warfare the only thing they can do with since the only thing they can do with since the only thing they can do with since the back against the war is to keep the back against the war is to keep the back against the war is to keep escalating escalating escalating and but and the West has put itself in and but and the West has put itself in and but and the West has put itself in the in the situation now we're sort of

the in the situation now we're sort of the in the situation now we're sort of already committed that the drive rush already committed that the drive rush already committed that the drive rush out so the only option the West has is out so the only option the West has is out so the only option the West has is to call Russ's Bluff and keep sending in to call Russ's Bluff and keep sending in to call Russ's Bluff and keep sending in more weapons more weapons more weapons this really bothers me because moloch this really bothers me because moloch this really bothers me because moloch can sometimes Drive competing parties to can sometimes Drive competing parties to can sometimes Drive competing parties to do something which is ultimately just do something which is ultimately just do something which is ultimately just really bad for both of them really bad for both of them really bad for both of them and uh you know what makes me even more and uh you know what makes me even more and uh you know what makes me even more worried is not just that I worried is not just that I worried is not just that I it's it's it's difficult to see an ending difficult to see an ending difficult to see an ending a quick peaceful ending to this tragedy a quick peaceful ending to this tragedy a quick peaceful ending to this tragedy that doesn't involve some horrible that doesn't involve some horrible that doesn't involve some horrible escalation escalation escalation um but also that we understand more um but also that we understand more um but also that we understand more clearly now just how horrible it was clearly now just how horrible it was clearly now just how horrible it was gonna it would be gonna it would be gonna it would be there was an amazing paper that was there was an amazing paper that was there was an amazing paper that was published in nature food this uh August published in nature food this uh August published in nature food this uh August by some of the top researchers who've by some of the top researchers who've by some of the top researchers who've been studying nuclear winter for a long been studying nuclear winter for a long been studying nuclear winter for a long time and what they basically did was time and what they basically did was time and what they basically did was they combined they combined they combined climate models climate models climate models with with with food agricultural models so instead of food agricultural models so instead of food agricultural models so instead of just saying yeah you know it gets really just saying yeah you know it gets really just saying yeah you know it gets really cold blah blah they figured out actually cold blah blah they figured out actually cold blah blah they figured out actually how many people would die in the how many people would die in the how many people would die in the different different countries different different countries different different countries and it's uh it's pretty mind-blowing you and it's uh it's pretty mind-blowing you and it's uh it's pretty mind-blowing you know so basically what happens you know know so basically what happens you know know so basically what happens you know is the thing that kills the most people is the thing that kills the most people is the thing that kills the most people is not the explosions it's not the is not the explosions it's not the is not the explosions it's not the radioactivity it's not the EMP Mayhem radioactivity it's not the EMP Mayhem radioactivity it's not the EMP Mayhem it's not the right it's not the right it's not the right the rampaging mobs forcing food no it's the rampaging mobs forcing food no it's the rampaging mobs forcing food no it's it's the fact that you get so much smoke it's the fact that you get so much smoke it's the fact that you get so much smoke coming up from the burning cities into coming up from the burning cities into coming up from the burning cities into the stratosphere that um the stratosphere that um the stratosphere that um that spreads around the earth from the

that spreads around the earth from the that spreads around the earth from the jet streams so jet streams so jet streams so in typical models you get like 10 years in typical models you get like 10 years in typical models you get like 10 years or so where it's just crazy cold and or so where it's just crazy cold and or so where it's just crazy cold and during the first year or after the the during the first year or after the the during the first year or after the the war and their models war and their models war and their models the temperature drops in in Nebraska and the temperature drops in in Nebraska and the temperature drops in in Nebraska and in the Ukraine bread baskets you know by in the Ukraine bread baskets you know by in the Ukraine bread baskets you know by like like like 20 Celsius or so if I remember 20 Celsius or so if I remember 20 Celsius or so if I remember no yeah 2030 Celsius depending on where no yeah 2030 Celsius depending on where no yeah 2030 Celsius depending on where you are 40 so it's just in some places you are 40 so it's just in some places you are 40 so it's just in some places which is you know 40 Fahrenheit to 80 which is you know 40 Fahrenheit to 80 which is you know 40 Fahrenheit to 80 Fahrenheit colder than what it would Fahrenheit colder than what it would Fahrenheit colder than what it would normally be normally be normally be so you know I'm not good at farming but so you know I'm not good at farming but so you know I'm not good at farming but uh it's knowing if it drops below uh it's knowing if it drops below uh it's knowing if it drops below freezing pretty much almost days in July freezing pretty much almost days in July freezing pretty much almost days in July and then like that's not good so they and then like that's not good so they and then like that's not good so they worked out they put this into their worked out they put this into their worked out they put this into their farming models and what they found was farming models and what they found was farming models and what they found was really interesting the countries that really interesting the countries that really interesting the countries that get the most hard hit are the ones in get the most hard hit are the ones in get the most hard hit are the ones in the northern hemisphere the northern hemisphere the northern hemisphere so in in the U.S so in in the U.S so in in the U.S and and one model they had about 99 of and and one model they had about 99 of and and one model they had about 99 of all Americans starving to death all Americans starving to death all Americans starving to death in Russia and China and Europe also in Russia and China and Europe also in Russia and China and Europe also about 99 98 starving to death about 99 98 starving to death about 99 98 starving to death so you might be like oh it's kind of so you might be like oh it's kind of so you might be like oh it's kind of poetic justice that both the Russians poetic justice that both the Russians poetic justice that both the Russians and the Americans and the Americans and the Americans 99 of them have to pay for it because it 99 of them have to pay for it because it 99 of them have to pay for it because it was their bombs that did it but you know was their bombs that did it but you know was their bombs that did it but you know that doesn't particularly cheer people that doesn't particularly cheer people that doesn't particularly cheer people up in Sweden or other random countries up in Sweden or other random countries up in Sweden or other random countries that have nothing to do with it right that have nothing to do with it right that have nothing to do with it right and um and um and um hit the hit the hit the I think it hasn't entered the mainstream

I think it hasn't entered the mainstream I think it hasn't entered the mainstream uh uh uh not understanding very much just like not understanding very much just like not understanding very much just like how bad this is uh most people how bad this is uh most people how bad this is uh most people especially a lot of people in decision especially a lot of people in decision especially a lot of people in decision making positions still think of nuclear making positions still think of nuclear making positions still think of nuclear weapons as something that makes you weapons as something that makes you weapons as something that makes you powerful powerful powerful uh scary powerful they don't think of it uh scary powerful they don't think of it uh scary powerful they don't think of it as something where uh yeah just as something where uh yeah just as something where uh yeah just to within a percent or two you know to within a percent or two you know to within a percent or two you know we're all just gonna starve to death we're all just gonna starve to death we're all just gonna starve to death and um and starving to death is and um and starving to death is and um and starving to death is is uh is uh is uh the worst way to die as Audemars all all the worst way to die as Audemars all all the worst way to die as Audemars all all the famines in history show the famines in history show the famines in history show the torture involved in that probably the torture involved in that probably the torture involved in that probably brings out the worst in people also when brings out the worst in people also when brings out the worst in people also when when people are desperate like this it's when people are desperate like this it's when people are desperate like this it's not so some people I've heard some not so some people I've heard some not so some people I've heard some people say that people say that people say that if that's what's going to happen there'd if that's what's going to happen there'd if that's what's going to happen there'd rather be Ground Zero and just get rather be Ground Zero and just get rather be Ground Zero and just get vaporized you know vaporized you know vaporized you know and but uh and but uh and but uh so but I think people underestimate the so but I think people underestimate the so but I think people underestimate the risk of this because they they risk of this because they they risk of this because they they aren't afraid of Morlock they think oh aren't afraid of Morlock they think oh aren't afraid of Morlock they think oh it's just gonna be because humans don't it's just gonna be because humans don't it's just gonna be because humans don't want this so it's not going to happen want this so it's not going to happen want this so it's not going to happen that's the whole point the mark that that's the whole point the mark that that's the whole point the mark that things happened that nobody wanted and things happened that nobody wanted and things happened that nobody wanted and that applies to nuclear weapons and that that applies to nuclear weapons and that that applies to nuclear weapons and that applies to AGI applies to AGI applies to AGI exactly and it applies to some of the exactly and it applies to some of the exactly and it applies to some of the things that people have gotten most things that people have gotten most things that people have gotten most upset with capitalism for also right upset with capitalism for also right upset with capitalism for also right where everybody was just kind of Trapped where everybody was just kind of Trapped where everybody was just kind of Trapped you know

you know you know it's not to see if some company does it's not to see if some company does it's not to see if some company does something it causes a lot of harm and not that the it causes a lot of harm and not that the CEO is a bad person but she or he knew CEO is a bad person but she or he knew CEO is a bad person but she or he knew that you know that the other all the that you know that the other all the that you know that the other all the other companies were doing this too so other companies were doing this too so other companies were doing this too so Morlock is um as a former foe I hope for someone as a former foe I hope for someone making making making good movies so we can see who the real good movies so we can see who the real good movies so we can see who the real enemy is so we don't because we're not enemy is so we don't because we're not enemy is so we don't because we're not fighting against each other molok makes fighting against each other molok makes fighting against each other molok makes us fight against each other that small us fight against each other that small us fight against each other that small that's what Malik's superpower is that's what Malik's superpower is that's what Malik's superpower is The Hope here is any kind of technology The Hope here is any kind of technology The Hope here is any kind of technology or other mechanism that lets us instead or other mechanism that lets us instead or other mechanism that lets us instead realize realize realize that we're finding The Wrong Enemy right that we're finding The Wrong Enemy right that we're finding The Wrong Enemy right now it's such a fascinating battle it's now it's such a fascinating battle it's now it's such a fascinating battle it's not a sport system it's US versus it not a sport system it's US versus it not a sport system it's US versus it yeah yeah we are fighting Malik for yeah yeah we are fighting Malik for yeah yeah we are fighting Malik for human survival we as a civilization have human survival we as a civilization have human survival we as a civilization have you seen the movie Needful Things you seen the movie Needful Things you seen the movie Needful Things it's a Stephen King novel I love Stephen it's a Stephen King novel I love Stephen it's a Stephen King novel I love Stephen King and uh Max fonseedov Swedish actors King and uh Max fonseedov Swedish actors King and uh Max fonseedov Swedish actors playing the guys it's brilliant exactly playing the guys it's brilliant exactly playing the guys it's brilliant exactly I just thought I hadn't thought about I just thought I hadn't thought about I just thought I hadn't thought about that until now but that's the closest that until now but that's the closest that until now but that's the closest I've seen to a a movie about moloch I I've seen to a a movie about moloch I I've seen to a a movie about moloch I don't want to spoil the film for anyone don't want to spoil the film for anyone don't want to spoil the film for anyone who wants to watch it but basically who wants to watch it but basically who wants to watch it but basically it's about this guy who turns out that it's about this guy who turns out that it's about this guy who turns out that you can interpret him as the devil or you can interpret him as the devil or you can interpret him as the devil or whatever but he doesn't actually ever go whatever but he doesn't actually ever go whatever but he doesn't actually ever go around and kill people or torture people around and kill people or torture people around and kill people or torture people with burning coal or anything

with burning coal or anything with burning coal or anything he makes everybody fight each other it he makes everybody fight each other it he makes everybody fight each other it makes everybody hate fear each other makes everybody hate fear each other makes everybody hate fear each other hate each other and then kill each other hate each other and then kill each other hate each other and then kill each other so that that's the movie about malloc so that that's the movie about malloc so that that's the movie about malloc you know love is the answer that seems you know love is the answer that seems you know love is the answer that seems to be to be to be um um um one of the ways to fight one of the ways to fight one of the ways to fight Malik is by Malik is by Malik is by um compassion by seeing the common um compassion by seeing the common um compassion by seeing the common Humanity yes yes and to not sound so we Humanity yes yes and to not sound so we Humanity yes yes and to not sound so we don't sound like like uh what's the don't sound like like uh what's the don't sound like like uh what's the Kumbaya Tree Huggers here right Kumbaya Tree Huggers here right Kumbaya Tree Huggers here right we're not just saying love and peace man we're not just saying love and peace man we're not just saying love and peace man we're we're trying to actually help we're we're trying to actually help we're we're trying to actually help people understand the true facts people understand the true facts people understand the true facts about the other side about the other side about the other side and feel the compassion and feel the compassion and feel the compassion because because because the truth makes you more compassionate the truth makes you more compassionate the truth makes you more compassionate right right right so I I that's why I really like using AI so I I that's why I really like using AI so I I that's why I really like using AI for truth and for truth seeking for truth and for truth seeking for truth and for truth seeking Technologies can that can as a result you know get us that can as a result you know get us more love than hate and more love than hate and more love than hate and and even if you can't get love you know and even if you can't get love you know and even if you can't get love you know settle for settle for some understanding settle for settle for some understanding settle for settle for some understanding which already gives compassion if which already gives compassion if which already gives compassion if someone is like you know someone is like you know someone is like you know I really disagree with you I really disagree with you I really disagree with you Lex but I can see why you're where Lex but I can see why you're where Lex but I can see why you're where you're coming from you're not a you're coming from you're not a you're coming from you're not a bad person who needs to be destroyed but bad person who needs to be destroyed but bad person who needs to be destroyed but I disagree with you and I'm happy to I disagree with you and I'm happy to I disagree with you and I'm happy to have an argument about it you know have an argument about it you know have an argument about it you know that's a lot of progress compared to

that's a lot of progress compared to that's a lot of progress compared to where we are 2023 in the public space where we are 2023 in the public space where we are 2023 in the public space wouldn't you say wouldn't you say wouldn't you say if we solve the AI safety problem as if we solve the AI safety problem as if we solve the AI safety problem as we've talked about and then you Max tag we've talked about and then you Max tag we've talked about and then you Max tag Mark who has been talking about this Mark who has been talking about this Mark who has been talking about this uh for many years get to sit down with uh for many years get to sit down with uh for many years get to sit down with the AGI with the early AGI system on a the AGI with the early AGI system on a the AGI with the early AGI system on a beach with a drink beach with a drink beach with a drink what kind of what would you ask her what what kind of what would you ask her what what kind of what would you ask her what kind of question would you ask what kind of question would you ask what kind of question would you ask what would you talk about would you talk about would you talk about something so much smarter than you a really Zinger of a question that's a a really Zinger of a question that's a good one would you be afraid good one would you be afraid good one would you be afraid to ask some questions no to ask some questions no to ask some questions no see I'm not afraid of the truth I'm very see I'm not afraid of the truth I'm very see I'm not afraid of the truth I'm very humble I know I'm just a meat pie you're humble I know I'm just a meat pie you're humble I know I'm just a meat pie you're with all these flaws you know but yeah I with all these flaws you know but yeah I with all these flaws you know but yeah I I have that we talked a lot about Homo I have that we talked a lot about Homo I have that we talked a lot about Homo sentience I really already tried that sentience I really already tried that sentience I really already tried that for a long time with myself just so that for a long time with myself just so that for a long time with myself just so that is what's really valuable about being is what's really valuable about being is what's really valuable about being alive for me is that I have these alive for me is that I have these alive for me is that I have these meaningful experiences meaningful experiences meaningful experiences it's not that I'm it's not that I'm it's not that I'm have what I'm good at this or good at have what I'm good at this or good at have what I'm good at this or good at that or whatever because there's so much that or whatever because there's so much that or whatever because there's so much I suck at then I suck at then I suck at then so you're not afraid for the system to so you're not afraid for the system to so you're not afraid for the system to show you just how dumb you are no no in show you just how dumb you are no no in show you just how dumb you are no no in fact my son reminds me of that fact my son reminds me of that fact my son reminds me of that you could find out how dumb you are in you could find out how dumb you are in you could find out how dumb you are in terms of physics how little how little terms of physics how little how little terms of physics how little how little we humans understand I'm cool with that we humans understand I'm cool with that we humans understand I'm cool with that I think I think um I think I think um I think I think um so I I can't waffle my way out of this so I I can't waffle my way out of this so I I can't waffle my way out of this question it's a fair one it was tough

question it's a fair one it was tough question it's a fair one it was tough I think given that I'm a really really I think given that I'm a really really I think given that I'm a really really curious person that's curious person that's curious person that's really the defining part of who I am I'm really the defining part of who I am I'm really the defining part of who I am I'm so curious uh I have some Physics questions uh I have some Physics questions I loved I love to understand I have some I loved I love to understand I have some I loved I love to understand I have some questions about Consciousness about the questions about Consciousness about the questions about Consciousness about the nature of reality or just really really nature of reality or just really really nature of reality or just really really love to understand also love to understand also love to understand also I can tell you one for example that I've I can tell you one for example that I've I can tell you one for example that I've been obsessing about a lot recently so I believe that so I believe that it's supposed to Noni is right I suppose it's supposed to Noni is right I suppose it's supposed to Noni is right I suppose there are some information processing there are some information processing there are some information processing systems that are conscious and some are systems that are conscious and some are systems that are conscious and some are not suppose you can even make reasonably not suppose you can even make reasonably not suppose you can even make reasonably smart things like gpt4 that are not smart things like gpt4 that are not smart things like gpt4 that are not conscious but you can also make them conscious but you can also make them conscious but you can also make them conscious conscious conscious here's the question that keeps me make here's the question that keeps me make here's the question that keeps me make it might is it the case that the unconscious is it the case that the unconscious zombie systems that are really zombie systems that are really zombie systems that are really intelligent are also really efficient so intelligent are also really efficient so intelligent are also really efficient so they're really inefficient they're really inefficient they're really inefficient so that when you try to make things more so that when you try to make things more so that when you try to make things more efficient we still naturally be a efficient we still naturally be a efficient we still naturally be a pressure to do they become conscious pressure to do they become conscious pressure to do they become conscious I'm kind of hoping that that's correct I'm kind of hoping that that's correct I'm kind of hoping that that's correct and I do you want me to give you a hand and I do you want me to give you a hand and I do you want me to give you a hand wavy argument for it you know like in my lab again every time we look at in my lab again every time we look at how it how these large language models how it how these large language models how it how these large language models do something we see that they do them in do something we see that they do them in do something we see that they do them in really dumb ways and you could make it really dumb ways and you could make it really dumb ways and you could make it make it better if if you uh

make it better if if you uh make it better if if you uh we have loops in our computer language we have loops in our computer language we have loops in our computer language for a reason for a reason for a reason the code which would get way way longer the code which would get way way longer the code which would get way way longer if you weren't allowed to use them it's if you weren't allowed to use them it's if you weren't allowed to use them it's more efficient to have the loops more efficient to have the loops more efficient to have the loops and and and in order to have self-reflection whether in order to have self-reflection whether in order to have self-reflection whether it's conscious or not right even an it's conscious or not right even an it's conscious or not right even an operating system knows things about operating system knows things about operating system knows things about itself right itself right itself right you need to have loops already and so I you need to have loops already and so I you need to have loops already and so I think this is I'm waving my hands a lot think this is I'm waving my hands a lot think this is I'm waving my hands a lot but I suspect that um but I suspect that um but I suspect that um the most efficient way of implementing a the most efficient way of implementing a the most efficient way of implementing a given level of intelligence given level of intelligence given level of intelligence has Loops in it the self-reflection it has Loops in it the self-reflection it has Loops in it the self-reflection it can can can and will be conscious isn't that great and will be conscious isn't that great and will be conscious isn't that great news yes if it's true it's wonderful news yes if it's true it's wonderful news yes if it's true it's wonderful because then we don't have to fear the because then we don't have to fear the because then we don't have to fear the ultimate zombie apocalypse and I think ultimate zombie apocalypse and I think ultimate zombie apocalypse and I think if you look at our brains actually our if you look at our brains actually our if you look at our brains actually our brains are part zombie brains are part zombie brains are part zombie and part conscious when I open my eyes when I open my eyes I immediately I immediately I immediately take all these pixels and hit my ref on take all these pixels and hit my ref on take all these pixels and hit my ref on my retina right and like oh that's Lex my retina right and like oh that's Lex my retina right and like oh that's Lex but I have no freaking clue of how I did but I have no freaking clue of how I did but I have no freaking clue of how I did that computation it's actually quite that computation it's actually quite that computation it's actually quite complicated right it was only relatively complicated right it was only relatively complicated right it was only relatively recently we could even do it well with recently we could even do it well with recently we could even do it well with machines right you get a bunch of machines right you get a bunch of machines right you get a bunch of information processing happening in my information processing happening in my information processing happening in my retina and then it goes to the lateral retina and then it goes to the lateral retina and then it goes to the lateral nucleus my Thalamus into the area V1 V2 nucleus my Thalamus into the area V1 V2 nucleus my Thalamus into the area V1 V2 V4 and the fusiform face area here that

V4 and the fusiform face area here that V4 and the fusiform face area here that Nancy can wish her MIT invented and blah Nancy can wish her MIT invented and blah Nancy can wish her MIT invented and blah blah blah blah blah blah blah I'd have no freaking clue how that I'd have no freaking clue how that I'd have no freaking clue how that worked right it feels to me subjectively worked right it feels to me subjectively worked right it feels to me subjectively like my conscious module like my conscious module like my conscious module just got a little email it's like face facial processing fit face facial processing fit task complete it's Lex yeah I'm gonna task complete it's Lex yeah I'm gonna task complete it's Lex yeah I'm gonna just go with that right so this fits just go with that right so this fits just go with that right so this fits perfectly with tanoni's model because perfectly with tanoni's model because perfectly with tanoni's model because this was all one way with the this was all one way with the this was all one way with the information processing mainly and uh it turned out for that particular and uh it turned out for that particular task that's all you needed and it task that's all you needed and it task that's all you needed and it probably was kind of the most efficient probably was kind of the most efficient probably was kind of the most efficient way to do it but there are a lot of way to do it but there are a lot of way to do it but there are a lot of other things that we associate with other things that we associate with other things that we associate with higher intelligence and planning and and higher intelligence and planning and and higher intelligence and planning and and so on and so forth where you kind of so on and so forth where you kind of so on and so forth where you kind of want to have loops and be able to want to have loops and be able to want to have loops and be able to ruminate and self-reflect and introspect ruminate and self-reflect and introspect ruminate and self-reflect and introspect and so on where and so on where and so on where my hunch is that if you want to fake my hunch is that if you want to fake my hunch is that if you want to fake that with a zombie system that just all that with a zombie system that just all that with a zombie system that just all goes one way you have to like unroll goes one way you have to like unroll goes one way you have to like unroll those loops and it just really really those loops and it just really really those loops and it just really really long and it's much more inefficient long and it's much more inefficient long and it's much more inefficient so I'm actually hopeful that AI if in so I'm actually hopeful that AI if in so I'm actually hopeful that AI if in the future we have all these various the future we have all these various the future we have all these various Sublime and interesting machines that do Sublime and interesting machines that do Sublime and interesting machines that do cool things cool things cool things and are aligned with that and are aligned with that and are aligned with that they will be at least they will also they will be at least they will also they will be at least they will also have Consciousness for the kind of these have Consciousness for the kind of these have Consciousness for the kind of these things that we do things that we do things that we do the great intelligence is also the great intelligence is also the great intelligence is also correlated to Great Consciousness or a correlated to Great Consciousness or a correlated to Great Consciousness or a deep kind of Consciousness yes so that's

deep kind of Consciousness yes so that's deep kind of Consciousness yes so that's a happy thought for me because the a happy thought for me because the a happy thought for me because the zombie of a couple of apocalypse really zombie of a couple of apocalypse really zombie of a couple of apocalypse really is my worst nightmare of all it would be is my worst nightmare of all it would be is my worst nightmare of all it would be like adding insult to injury not only like adding insult to injury not only like adding insult to injury not only did we get replaced but we freaking did we get replaced but we freaking did we get replaced but we freaking replaced ourselves by zombies like how replaced ourselves by zombies like how replaced ourselves by zombies like how dumb can we be that's such a beautiful dumb can we be that's such a beautiful dumb can we be that's such a beautiful vision and that's actually a provable vision and that's actually a provable vision and that's actually a provable one that's one that we humans can Intuit one that's one that we humans can Intuit one that's one that we humans can Intuit and prove that those two things are and prove that those two things are and prove that those two things are correlated as we start to understand correlated as we start to understand correlated as we start to understand what it means to be intelligent and what what it means to be intelligent and what what it means to be intelligent and what it means to be conscious which these it means to be conscious which these it means to be conscious which these systems uh early agi-like systems will systems uh early agi-like systems will systems uh early agi-like systems will help us understand and I just want to help us understand and I just want to help us understand and I just want to say one more thing this is super say one more thing this is super say one more thing this is super important most of my colleagues when I important most of my colleagues when I important most of my colleagues when I started going on in my Consciousness started going on in my Consciousness started going on in my Consciousness tell me that it's all and I tell me that it's all and I tell me that it's all and I should stop talking about it should stop talking about it should stop talking about it I hear a little inner voice from my I hear a little inner voice from my I hear a little inner voice from my father and for my mom saying keep father and for my mom saying keep father and for my mom saying keep talking about it because I think they're talking about it because I think they're talking about it because I think they're wrong and and and wrong and and and wrong and and and the main the main the main way to convince people like that that way to convince people like that that way to convince people like that that they're wrong if they say that they're wrong if they say that they're wrong if they say that Consciousness is just equal to Consciousness is just equal to Consciousness is just equal to intelligence just ask them what's wrong intelligence just ask them what's wrong intelligence just ask them what's wrong with torture or why are You Against with torture or why are You Against with torture or why are You Against torture torture torture if it's just about you know if it's just about you know if it's just about you know these these particles are moving this these these particles are moving this these these particles are moving this way rather than that way way rather than that way way rather than that way and there is no such thing as subjective and there is no such thing as subjective and there is no such thing as subjective experience what's wrong with torture I experience what's wrong with torture I experience what's wrong with torture I mean do you have a good comeback to that mean do you have a good comeback to that mean do you have a good comeback to that no it seems like suffering suffering no it seems like suffering suffering no it seems like suffering suffering imposed onto other humans is somehow imposed onto other humans is somehow imposed onto other humans is somehow deeply wrong in a way that intelligence deeply wrong in a way that intelligence deeply wrong in a way that intelligence doesn't quite explain that if someone doesn't quite explain that if someone doesn't quite explain that if someone tells me well you know tells me well you know tells me well you know It's Just an Illusion Consciousness

It's Just an Illusion Consciousness It's Just an Illusion Consciousness whatever you know whatever you know whatever you know I would like to invite them the next I would like to invite them the next I would like to invite them the next time they're having surgery to do it time they're having surgery to do it time they're having surgery to do it without anesthesia without anesthesia without anesthesia what is anesthesia really doing if you what is anesthesia really doing if you what is anesthesia really doing if you have it you can have it local anesthesia have it you can have it local anesthesia have it you can have it local anesthesia when you're awake I have that when they when you're awake I have that when they when you're awake I have that when they fix my shoulder right it's super fix my shoulder right it's super fix my shoulder right it's super entertaining uh entertaining uh entertaining uh what was that that it did it just what was that that it did it just what was that that it did it just removed my subjective experience of pain removed my subjective experience of pain removed my subjective experience of pain it didn't change anything it was it didn't change anything it was it didn't change anything it was actually happening in my shoulder right actually happening in my shoulder right actually happening in my shoulder right so if someone says that's all so if someone says that's all so if someone says that's all and skip the anesthesia as my advice and skip the anesthesia as my advice and skip the anesthesia as my advice this is incredibly Central this is incredibly Central this is incredibly Central it could be fundamental to whatever this it could be fundamental to whatever this it could be fundamental to whatever this thing we have going on here it is thing we have going on here it is thing we have going on here it is fundamental because we're we what we fundamental because we're we what we fundamental because we're we what we feel is so fundamental is suffering and feel is so fundamental is suffering and feel is so fundamental is suffering and joy and pleasure and meaning and joy and pleasure and meaning and joy and pleasure and meaning and that's all those are all subjective that's all those are all subjective that's all those are all subjective experiences there experiences there experiences there and let's not those are the elephant in and let's not those are the elephant in and let's not those are the elephant in the room that's what makes life worth the room that's what makes life worth the room that's what makes life worth living and that's what can make it living and that's what can make it living and that's what can make it horrible if it's just a bunch of horrible if it's just a bunch of horrible if it's just a bunch of suffering so let's not make a mistake of suffering so let's not make a mistake of suffering so let's not make a mistake of saying that that's all saying that that's all saying that that's all and let's not make the mistake of uh and let's not make the mistake of uh and let's not make the mistake of uh not instilling the AI systems with that not instilling the AI systems with that not instilling the AI systems with that same same same thing that makes us uh special yeah thing that makes us uh special yeah thing that makes us uh special yeah Max uh it's a huge honor that you will Max uh it's a huge honor that you will Max uh it's a huge honor that you will sit down to me the first time sit down to me the first time sit down to me the first time uh on the first episode of this podcast uh on the first episode of this podcast uh on the first episode of this podcast it's a huge honest sit down with me it's a huge honest sit down with me it's a huge honest sit down with me again and talk about this what I think again and talk about this what I think again and talk about this what I think is uh the most important topic

is uh the most important topic is uh the most important topic the most important problem that we the most important problem that we the most important problem that we humans have to face and hopefully solve humans have to face and hopefully solve humans have to face and hopefully solve yeah well the honor is all mine and I'm yeah well the honor is all mine and I'm yeah well the honor is all mine and I'm so grateful to Youth for making more so grateful to Youth for making more so grateful to Youth for making more people aware of the fact that Humanity people aware of the fact that Humanity people aware of the fact that Humanity has reached the most important fork in has reached the most important fork in has reached the most important fork in the road ever and it's history and the road ever and it's history and the road ever and it's history and that's that's that's turn in the correct direction turn in the correct direction turn in the correct direction thanks for listening to this thanks for listening to this thanks for listening to this conversation with Max tagmark to support conversation with Max tagmark to support conversation with Max tagmark to support this podcast please check out our this podcast please check out our this podcast please check out our sponsors in the description and now let sponsors in the description and now let sponsors in the description and now let me leave you some words from Frank me leave you some words from Frank me leave you some words from Frank Herbert Herbert Herbert history is a constant race between history is a constant race between history is a constant race between invention and catastrophe invention and catastrophe invention and catastrophe thank you for listening and hope to see thank you for listening and hope to see thank you for listening and hope to see you next time