Roman Yampolskiy: Dangers of Superintelligent AI | Lex Fridman Podcast #431

if we create General super intelligences if we create General super intelligences I don't see a good outcome longterm for I don't see a good outcome longterm for I don't see a good outcome longterm for Humanity so that is X risk existential Humanity so that is X risk existential Humanity so that is X risk existential risk everyone's dead there is srisk risk everyone's dead there is srisk risk everyone's dead there is srisk suffering risks where everyone wishes suffering risks where everyone wishes suffering risks where everyone wishes they were dead we have also idea for IR they were dead we have also idea for IR they were dead we have also idea for IR risk iyy risks where we lost our meaning risk iyy risks where we lost our meaning risk iyy risks where we lost our meaning the systems can be more creative they the systems can be more creative they the systems can be more creative they can do all the jobs it's not obvious can do all the jobs it's not obvious can do all the jobs it's not obvious what you have to contribute to a world what you have to contribute to a world what you have to contribute to a world where super intelligence exists of where super intelligence exists of where super intelligence exists of course you can have all the variants you course you can have all the variants you course you can have all the variants you mentioned where we are safe we are kept mentioned where we are safe we are kept mentioned where we are safe we are kept alive but we are not in control we are alive but we are not in control we are alive but we are not in control we are not deciding anything we like animals in not deciding anything we like animals in not deciding anything we like animals in a zo there is a zo there is a zo there is again possibilities we can come up with again possibilities we can come up with again possibilities we can come up with as very smart humans and then as very smart humans and then as very smart humans and then possibility is something a thousand possibility is something a thousand possibility is something a thousand times smarter can come up with for times smarter can come up with for times smarter can come up with for reasons we cannot reasons we cannot reasons we cannot comprehend the following is a comprehend the following is a comprehend the following is a conversation with Roman yski an AI conversation with Roman yski an AI conversation with Roman yski an AI Safety and Security research Safety and Security research Safety and Security research and author of a new book titled AI and author of a new book titled AI and author of a new book titled AI unexplainable unpredictable unexplainable unpredictable unexplainable unpredictable uncontrollable he argues that there's uncontrollable he argues that there's uncontrollable he argues that there's almost 100% chance that AGI will almost 100% chance that AGI will almost 100% chance that AGI will eventually destroy human civilization as eventually destroy human civilization as eventually destroy human civilization as an aside let me say that I will have an aside let me say that I will have an aside let me say that I will have many often technical conversations on many often technical conversations on many often technical conversations on the topic of AI often with Engineers the topic of AI often with Engineers the topic of AI often with Engineers building the state of the art AI systems building the state of the art AI systems building the state of the art AI systems I would say those folks put the infamous I would say those folks put the infamous I would say those folks put the infamous P Doom or the probability of a GI P Doom or the probability of a GI P Doom or the probability of a GI killing all humans at around 1 to killing all humans at around 1 to killing all humans at around 1 to 20% but it's also important to talk to

20% but it's also important to talk to 20% but it's also important to talk to folks who put that value at 70 80 folks who put that value at 70 80 folks who put that value at 70 80 90 and is in the case of Roman at 90 and is in the case of Roman at 90 and is in the case of Roman at 99.99 and many more 9es 99.99 and many more 9es 99.99 and many more 9es per. I'm personally excited for the per. I'm personally excited for the per. I'm personally excited for the future and believe it will be a good one future and believe it will be a good one future and believe it will be a good one in part because of the amazing in part because of the amazing in part because of the amazing technological innovation we humans technological innovation we humans technological innovation we humans create but we must absolutely not do so create but we must absolutely not do so create but we must absolutely not do so with blinders on ignoring the possible with blinders on ignoring the possible with blinders on ignoring the possible risks including existential risks of risks including existential risks of risks including existential risks of those those those Technologies that's what this Technologies that's what this Technologies that's what this conversation is about this is the Lex conversation is about this is the Lex conversation is about this is the Lex Freedman podcast to support it please Freedman podcast to support it please Freedman podcast to support it please check out our sponsors in the check out our sponsors in the check out our sponsors in the description and now dear friends here's description and now dear friends here's description and now dear friends here's Roman Roman Roman yski what to you is the probability that yski what to you is the probability that yski what to you is the probability that super intelligent AI will destroy all super intelligent AI will destroy all super intelligent AI will destroy all human civilization what's the time frame human civilization what's the time frame human civilization what's the time frame let's say 100 years in the next 100 let's say 100 years in the next 100 let's say 100 years in the next 100 years so the problem of controlling AI years so the problem of controlling AI years so the problem of controlling AI or super intelligence in my opinion is or super intelligence in my opinion is or super intelligence in my opinion is like a problem of creating a Perpetual like a problem of creating a Perpetual like a problem of creating a Perpetual safety Machine by analogy with perpetual safety Machine by analogy with perpetual safety Machine by analogy with perpetual motion machine is impossible yeah we may motion machine is impossible yeah we may motion machine is impossible yeah we may succeed and do a good job with GPT 5 6 7 succeed and do a good job with GPT 5 6 7 succeed and do a good job with GPT 5 6 7 but they just keep improving learning but they just keep improving learning but they just keep improving learning eventually self-modifying interacting eventually self-modifying interacting eventually self-modifying interacting with the environment interacting with with the environment interacting with with the environment interacting with malevolent malevolent malevolent actors the difference between cyber actors the difference between cyber actors the difference between cyber security narrow AI safety and safety for

security narrow AI safety and safety for security narrow AI safety and safety for General AI for super intelligence is General AI for super intelligence is General AI for super intelligence is that we don't get a second chance with that we don't get a second chance with that we don't get a second chance with cyber security somebody hacks your cyber security somebody hacks your cyber security somebody hacks your account what's the big deal you get a account what's the big deal you get a account what's the big deal you get a new password new credit card you move new password new credit card you move new password new credit card you move on here if we're talking about on here if we're talking about on here if we're talking about existential risks you only get one existential risks you only get one existential risks you only get one chance so you're really asking me what chance so you're really asking me what chance so you're really asking me what are the chances that will create the are the chances that will create the are the chances that will create the most complex software ever on the first most complex software ever on the first most complex software ever on the first try with zero bugs and it will continue try with zero bugs and it will continue try with zero bugs and it will continue have zero bugs for 100 years or have zero bugs for 100 years or have zero bugs for 100 years or more so there is an incremental more so there is an incremental more so there is an incremental Improvement of systems leading up to AGI Improvement of systems leading up to AGI Improvement of systems leading up to AGI to you it doesn't matter if we can keep to you it doesn't matter if we can keep to you it doesn't matter if we can keep those safe there's going to be one level those safe there's going to be one level those safe there's going to be one level of of of system at which you cannot possibly system at which you cannot possibly system at which you cannot possibly control control control it I don't think we so far have made any it I don't think we so far have made any it I don't think we so far have made any system safe at the level of capability system safe at the level of capability system safe at the level of capability they display they already have made they display they already have made they display they already have made mistakes we had accidents they've been mistakes we had accidents they've been mistakes we had accidents they've been jailbroken I don't think there is a jailbroken I don't think there is a jailbroken I don't think there is a single large language model today which single large language model today which single large language model today which no one was successful at making do no one was successful at making do no one was successful at making do something developers didn't intend it to something developers didn't intend it to something developers didn't intend it to do but there's a difference between do but there's a difference between do but there's a difference between getting it to do something unintended getting it to do something unintended getting it to do something unintended getting it to do something that's getting it to do something that's getting it to do something that's painful costly destructive and something painful costly destructive and something painful costly destructive and something that's destructive to the level of that's destructive to the level of that's destructive to the level of hurting billions of people or hundreds hurting billions of people or hundreds hurting billions of people or hundreds of millions of people billions of people of millions of people billions of people of millions of people billions of people or the entirety of human civilization or the entirety of human civilization or the entirety of human civilization that's a big leap exactly but the

that's a big leap exactly but the that's a big leap exactly but the systems we have today have capability of systems we have today have capability of systems we have today have capability of causing x amount of damage so then they causing x amount of damage so then they causing x amount of damage so then they fail that's all we get if we develop fail that's all we get if we develop fail that's all we get if we develop systems capable of impacting all of systems capable of impacting all of systems capable of impacting all of humanity all of universe the damage is humanity all of universe the damage is humanity all of universe the damage is proportionate what to you are the proportionate what to you are the proportionate what to you are the possible ways that possible ways that possible ways that such kind of mass murder of humans can such kind of mass murder of humans can such kind of mass murder of humans can happen it's always a wonderful question happen it's always a wonderful question happen it's always a wonderful question so one of the chapters in my new book is so one of the chapters in my new book is so one of the chapters in my new book is about unpredictability I argue that we about unpredictability I argue that we about unpredictability I argue that we cannot predict what a smarter system cannot predict what a smarter system cannot predict what a smarter system will do so you're really not asking me will do so you're really not asking me will do so you're really not asking me how super intelligence will kill how super intelligence will kill how super intelligence will kill everyone you're asking me how I would do everyone you're asking me how I would do everyone you're asking me how I would do it and I think it's not that interesting it and I think it's not that interesting it and I think it's not that interesting I can tell you about the standard you I can tell you about the standard you I can tell you about the standard you know nanotag synthetic bionuclear super know nanotag synthetic bionuclear super know nanotag synthetic bionuclear super intelligence will come up with something intelligence will come up with something intelligence will come up with something completely new completely completely new completely completely new completely super we may not even recognize that as super we may not even recognize that as super we may not even recognize that as a possible path to achieve that goal so a possible path to achieve that goal so a possible path to achieve that goal so there is like a unlimited level of there is like a unlimited level of there is like a unlimited level of creativity in terms of how humans could creativity in terms of how humans could creativity in terms of how humans could be be be killed but you know we could still killed but you know we could still killed but you know we could still investigate possible ways of doing it investigate possible ways of doing it investigate possible ways of doing it not how to do it but the at the end what not how to do it but the at the end what not how to do it but the at the end what is the methodology that does it you know is the methodology that does it you know is the methodology that does it you know shutting off the shutting off the shutting off the power and then humans start killing each power and then humans start killing each power and then humans start killing each other maybe because the resource is are other maybe because the resource is are other maybe because the resource is are really constrained that there then really constrained that there then really constrained that there then there's the actual use of weapons like there's the actual use of weapons like there's the actual use of weapons like nuclear weapons or developing artificial nuclear weapons or developing artificial nuclear weapons or developing artificial pathogens viruses that kind of stuff we

pathogens viruses that kind of stuff we pathogens viruses that kind of stuff we could still kind of think through that could still kind of think through that could still kind of think through that and defend against it right there's a and defend against it right there's a and defend against it right there's a ceiling to the creativity of mass murder ceiling to the creativity of mass murder ceiling to the creativity of mass murder of humans here right the options are of humans here right the options are of humans here right the options are limited they are limited by how limited they are limited by how limited they are limited by how imaginative we are if you are that much imaginative we are if you are that much imaginative we are if you are that much smarter that much more creative you are smarter that much more creative you are smarter that much more creative you are capable of thinking across multiple capable of thinking across multiple capable of thinking across multiple domains do no research in physics and domains do no research in physics and domains do no research in physics and biology you may not be limited by those biology you may not be limited by those biology you may not be limited by those tools if squirrels were planning to kill tools if squirrels were planning to kill tools if squirrels were planning to kill humans they would have a set of possible humans they would have a set of possible humans they would have a set of possible ways of doing it but they would never ways of doing it but they would never ways of doing it but they would never consider things we can come up so are consider things we can come up so are consider things we can come up so are you are you thinking about mass murder you are you thinking about mass murder you are you thinking about mass murder and destruction of human civilization or and destruction of human civilization or and destruction of human civilization or you thinking of with squirrels you put you thinking of with squirrels you put you thinking of with squirrels you put them in a zoo and they don't really know them in a zoo and they don't really know them in a zoo and they don't really know they're in a zoo if we just look at the they're in a zoo if we just look at the they're in a zoo if we just look at the entire set of undesirable entire set of undesirable entire set of undesirable trajectories majority of them are not trajectories majority of them are not trajectories majority of them are not going to be death most of them are going going to be death most of them are going going to be death most of them are going to be just like to be just like to be just like uh things like Brave New World uh things like Brave New World uh things like Brave New World where you know the squirrels are fed where you know the squirrels are fed where you know the squirrels are fed dopamine and they're all like doing some dopamine and they're all like doing some dopamine and they're all like doing some kind of fun activity and the sort of the kind of fun activity and the sort of the kind of fun activity and the sort of the fire the soul of humanity is lost fire the soul of humanity is lost fire the soul of humanity is lost because of the drug that's fed to it or because of the drug that's fed to it or because of the drug that's fed to it or like literally in a zoo we're in a zoo like literally in a zoo we're in a zoo like literally in a zoo we're in a zoo we're doing our thing we're like playing we're doing our thing we're like playing we're doing our thing we're like playing a game of a game of a game of Sims and the actual players playing that Sims and the actual players playing that Sims and the actual players playing that game are AI systems those are all game are AI systems those are all game are AI systems those are all undesirable because sort of sort of the undesirable because sort of sort of the undesirable because sort of sort of the the free the free the free will the fire of human consciousness is will the fire of human consciousness is will the fire of human consciousness is dimmed through that process but it's not dimmed through that process but it's not dimmed through that process but it's not killing humans so like are you thinking killing humans so like are you thinking killing humans so like are you thinking about that or is the biggest concern

about that or is the biggest concern about that or is the biggest concern literally the extinctions of humans I literally the extinctions of humans I literally the extinctions of humans I think about a lot of things so there is think about a lot of things so there is think about a lot of things so there is X risk existential risk everyone's dead X risk existential risk everyone's dead X risk existential risk everyone's dead there is srisk suffering risks where there is srisk suffering risks where there is srisk suffering risks where everyone wishes they were that we have everyone wishes they were that we have everyone wishes they were that we have also idea for IR risk iy risks where also idea for IR risk iy risks where also idea for IR risk iy risks where lost their meaning the systems can be lost their meaning the systems can be lost their meaning the systems can be more creative they can do all the jobs more creative they can do all the jobs more creative they can do all the jobs it's not obvious what you have to it's not obvious what you have to it's not obvious what you have to contribute to a world where super contribute to a world where super contribute to a world where super intelligence exists of course you can intelligence exists of course you can intelligence exists of course you can have all the variants you mentioned have all the variants you mentioned have all the variants you mentioned where we are safe we are kept alive but where we are safe we are kept alive but where we are safe we are kept alive but we are not in control we are not we are not in control we are not we are not in control we are not deciding anything we like animals in a deciding anything we like animals in a deciding anything we like animals in a zo there is zo there is zo there is again possibilities we can come up with again possibilities we can come up with again possibilities we can come up with as very smart humans and then as very smart humans and then as very smart humans and then possibility is something a thousand possibility is something a thousand possibility is something a thousand smarter can come up with for reasons we smarter can come up with for reasons we smarter can come up with for reasons we cannot comprehend I would love to sort cannot comprehend I would love to sort cannot comprehend I would love to sort of dig into each of those X risk srisk of dig into each of those X risk srisk of dig into each of those X risk srisk and IR risk so can can you like Linger and IR risk so can can you like Linger and IR risk so can can you like Linger on irisk what is that so Japanese on irisk what is that so Japanese on irisk what is that so Japanese concept of viky guy you find something concept of viky guy you find something concept of viky guy you find something which allows you to make money you are which allows you to make money you are which allows you to make money you are good at it and the society says we need good at it and the society says we need good at it and the society says we need it so like you have this awesome job you it so like you have this awesome job you it so like you have this awesome job you are are are podcaster gives you a lot of meaning you podcaster gives you a lot of meaning you podcaster gives you a lot of meaning you have a good life I assume you happy have a good life I assume you happy have a good life I assume you happy mhm that's what we want most people to mhm that's what we want most people to mhm that's what we want most people to find to have for many find to have for many find to have for many intellectuals it is their occupation intellectuals it is their occupation intellectuals it is their occupation which gives them a lot of meaning I am a which gives them a lot of meaning I am a which gives them a lot of meaning I am a researcher philosopher scholar that

researcher philosopher scholar that researcher philosopher scholar that means something to me in a world where means something to me in a world where means something to me in a world where an artist is not feeling appreciated an artist is not feeling appreciated an artist is not feeling appreciated because his art is just not competitive because his art is just not competitive because his art is just not competitive with what is produced by machines or with what is produced by machines or with what is produced by machines or writer or scientist will lose a lot of writer or scientist will lose a lot of writer or scientist will lose a lot of that that that and at the lower level we're talking and at the lower level we're talking and at the lower level we're talking about complete technological about complete technological about complete technological unemployment we're not losing 10% of unemployment we're not losing 10% of unemployment we're not losing 10% of jobs we're losing all jobs what do jobs we're losing all jobs what do jobs we're losing all jobs what do people do with all that free time what people do with all that free time what people do with all that free time what happens then everything Society is built happens then everything Society is built happens then everything Society is built on is completely modified in one on is completely modified in one on is completely modified in one generation it's not a slow process where generation it's not a slow process where generation it's not a slow process where we get to kind of figure out how to live we get to kind of figure out how to live we get to kind of figure out how to live that new lifestyle but it's uh pretty that new lifestyle but it's uh pretty that new lifestyle but it's uh pretty quick in that world can't humans just do quick in that world can't humans just do quick in that world can't humans just do what humans currently do with chess play what humans currently do with chess play what humans currently do with chess play each other have tournaments even though each other have tournaments even though each other have tournaments even though AI systems are far superior at this time AI systems are far superior at this time AI systems are far superior at this time in chess so we just create artificial in chess so we just create artificial in chess so we just create artificial games or for us they're real like the games or for us they're real like the games or for us they're real like the Olympics and we do all kinds of Olympics and we do all kinds of Olympics and we do all kinds of different competitions and have fun different competitions and have fun different competitions and have fun Focus maximize the fun and and uh let uh Focus maximize the fun and and uh let uh Focus maximize the fun and and uh let uh the AI focus on the productivity it's an the AI focus on the productivity it's an the AI focus on the productivity it's an option I have a paper where I try to option I have a paper where I try to option I have a paper where I try to solve the value alignment problem for solve the value alignment problem for solve the value alignment problem for multiple agents and the solution to multiple agents and the solution to multiple agents and the solution to avoid compromise is to give everyone a avoid compromise is to give everyone a avoid compromise is to give everyone a personal virtual Universe you can do personal virtual Universe you can do personal virtual Universe you can do whatever you want in that world you whatever you want in that world you whatever you want in that world you could be king you could be slave you could be king you could be slave you could be king you could be slave you decide what happens so it's basically a decide what happens so it's basically a decide what happens so it's basically a glorified video game where you get to

glorified video game where you get to glorified video game where you get to enjoy yourself and someone else takes enjoy yourself and someone else takes enjoy yourself and someone else takes care of your needs and the substrate care of your needs and the substrate care of your needs and the substrate alignment is the only thing we need to alignment is the only thing we need to alignment is the only thing we need to solve we don't have to get 8 billion solve we don't have to get 8 billion solve we don't have to get 8 billion humans to agree on anything mhm so okay humans to agree on anything mhm so okay humans to agree on anything mhm so okay so what why is that not a likely outcome so what why is that not a likely outcome so what why is that not a likely outcome why can't they systems create video why can't they systems create video why can't they systems create video games for us to lose ourselves in each games for us to lose ourselves in each games for us to lose ourselves in each each with an individual video game each with an individual video game each with an individual video game Universe some people say that's what Universe some people say that's what Universe some people say that's what happened we're in a simulation and we're happened we're in a simulation and we're happened we're in a simulation and we're playing that video game and now we're playing that video game and now we're playing that video game and now we're creating uh what maybe we're creating creating uh what maybe we're creating creating uh what maybe we're creating artificial threats for ourselves to be artificial threats for ourselves to be artificial threats for ourselves to be scared about cuz cuz fear is really scared about cuz cuz fear is really scared about cuz cuz fear is really exciting it allows us to play the video exciting it allows us to play the video exciting it allows us to play the video game more uh more vigorously and some game more uh more vigorously and some game more uh more vigorously and some people choose to play on a more people choose to play on a more people choose to play on a more difficult level with more con strange difficult level with more con strange difficult level with more con strange some say okay I'm just going to enjoy some say okay I'm just going to enjoy some say okay I'm just going to enjoy the game high privilege level absolutely the game high privilege level absolutely the game high privilege level absolutely so okay what was that paper on so okay what was that paper on so okay what was that paper on multi-agent value alignment personal multi-agent value alignment personal multi-agent value alignment personal universes personal universes so so universes personal universes so so universes personal universes so so that's one of the possible outcomes but that's one of the possible outcomes but that's one of the possible outcomes but what what what in general is the idea of what what what in general is the idea of what what what in general is the idea of the paper so it's looking at multiple the paper so it's looking at multiple the paper so it's looking at multiple agents they're human AI like a hybrid agents they're human AI like a hybrid agents they're human AI like a hybrid system whether it's humans and AI or is system whether it's humans and AI or is system whether it's humans and AI or is it looking at humans or just so this is it looking at humans or just so this is it looking at humans or just so this is intelligent agents in order to solve intelligent agents in order to solve intelligent agents in order to solve value alignment problem I'm trying to value alignment problem I'm trying to value alignment problem I'm trying to formalize it a little better usually formalize it a little better usually formalize it a little better usually we're talking about getting AIS to do we're talking about getting AIS to do we're talking about getting AIS to do what we want which is not well defined what we want which is not well defined what we want which is not well defined we're talking about creator of a system we're talking about creator of a system we're talking about creator of a system owner of that AI Humanity as a whole but owner of that AI Humanity as a whole but owner of that AI Humanity as a whole but we don't agree on much there is no

we don't agree on much there is no we don't agree on much there is no universally accepted ethics morals universally accepted ethics morals universally accepted ethics morals across cultures religions people have across cultures religions people have across cultures religions people have individually very different preferences individually very different preferences individually very different preferences politically and such so even if we politically and such so even if we politically and such so even if we somehow managed all the other aspects of somehow managed all the other aspects of somehow managed all the other aspects of it programming those fuzzy Concepts and it programming those fuzzy Concepts and it programming those fuzzy Concepts and getting to follow them closely we don't getting to follow them closely we don't getting to follow them closely we don't agree on what to program in so my agree on what to program in so my agree on what to program in so my solution was okay we don't have to solution was okay we don't have to solution was okay we don't have to compromise on room temperature you have compromise on room temperature you have compromise on room temperature you have your Universe I have mine whatever you your Universe I have mine whatever you your Universe I have mine whatever you want and if you like me you can invite want and if you like me you can invite want and if you like me you can invite me to visit your Universe we don't have me to visit your Universe we don't have me to visit your Universe we don't have to be independent but the point is you to be independent but the point is you to be independent but the point is you can be and virtual reality is getting can be and virtual reality is getting can be and virtual reality is getting pretty good it's going to hit a point pretty good it's going to hit a point pretty good it's going to hit a point where you can't tell the difference and where you can't tell the difference and where you can't tell the difference and if you can't tell if it's real or not if you can't tell if it's real or not if you can't tell if it's real or not what's the difference so basically give what's the difference so basically give what's the difference so basically give up on value alignment create and entire up on value alignment create and entire up on value alignment create and entire it's like the the Multiverse Theory this it's like the the Multiverse Theory this it's like the the Multiverse Theory this just create an entire universe for you just create an entire universe for you just create an entire universe for you where your values you still have to where your values you still have to where your values you still have to align with that individual they have to align with that individual they have to align with that individual they have to be happy in that simulation but it's a be happy in that simulation but it's a be happy in that simulation but it's a much easier problem to align with one much easier problem to align with one much easier problem to align with one agent versus 8 billion agents plus agent versus 8 billion agents plus agent versus 8 billion agents plus animals aliens so you convert the animals aliens so you convert the animals aliens so you convert the multi-agent problem into a single agent multi-agent problem into a single agent multi-agent problem into a single agent problem I'm trying to do that yeah okay problem I'm trying to do that yeah okay problem I'm trying to do that yeah okay is there any way to is so okay that's is there any way to is so okay that's is there any way to is so okay that's giving up on the on the value problem giving up on the on the value problem giving up on the on the value problem well is there any way to solve the value well is there any way to solve the value well is there any way to solve the value alignment problem where there's a bunch alignment problem where there's a bunch alignment problem where there's a bunch of humans multiple humans tens of humans of humans multiple humans tens of humans of humans multiple humans tens of humans or 8 billion humans that have very or 8 billion humans that have very or 8 billion humans that have very different set of values it seems different set of values it seems different set of values it seems contradictory I haven't seen anyone contradictory I haven't seen anyone contradictory I haven't seen anyone explain what it means outside of kind of

explain what it means outside of kind of explain what it means outside of kind of words which pack a lot make it good make words which pack a lot make it good make words which pack a lot make it good make it desirable make it something they it desirable make it something they it desirable make it something they don't regret but how do you specifically don't regret but how do you specifically don't regret but how do you specifically formalize those Notions how do you formalize those Notions how do you formalize those Notions how do you program women I haven't seen anyone uh program women I haven't seen anyone uh program women I haven't seen anyone uh make progress on that so far but isn't make progress on that so far but isn't make progress on that so far but isn't that the whole optimization Journey that that the whole optimization Journey that that the whole optimization Journey that we're doing as a human civilization we're doing as a human civilization we're doing as a human civilization we're looking at we're looking at we're looking at geopolitics nations are in a state of geopolitics nations are in a state of geopolitics nations are in a state of Anarchy with each other they start wars Anarchy with each other they start wars Anarchy with each other they start wars there's there's there's conflict and often times they have a conflict and often times they have a conflict and often times they have a very different views of what is good and very different views of what is good and very different views of what is good and and what is evil isn't that what we're and what is evil isn't that what we're and what is evil isn't that what we're trying to figure out just together trying to figure out just together trying to figure out just together trying to converge towards that so we're trying to converge towards that so we're trying to converge towards that so we're essentially trying to solve the value essentially trying to solve the value essentially trying to solve the value alignment problem with humans right but alignment problem with humans right but alignment problem with humans right but the examples you gave uh some of them the examples you gave uh some of them the examples you gave uh some of them are for example two different religions are for example two different religions are for example two different religions saying this is our holy sight and we are saying this is our holy sight and we are saying this is our holy sight and we are not willing to compromise it in any way not willing to compromise it in any way not willing to compromise it in any way if you can make two holy s sites in if you can make two holy s sites in if you can make two holy s sites in virtual world you solve the problem but virtual world you solve the problem but virtual world you solve the problem but if you only have one it's not divisible if you only have one it's not divisible if you only have one it's not divisible you kind of stuck there but what if we you kind of stuck there but what if we you kind of stuck there but what if we want to be a tension with each other and want to be a tension with each other and want to be a tension with each other and that through that tension we understand that through that tension we understand that through that tension we understand ourselves and we understand the world so ourselves and we understand the world so ourselves and we understand the world so that that's the intellectual Journey that that's the intellectual Journey that that's the intellectual Journey we're on we're on as a human we're on we're on as a human we're on we're on as a human civilization is we create intellectual civilization is we create intellectual civilization is we create intellectual and physical conflict and through that and physical conflict and through that and physical conflict and through that figure stuff out if we go back to that figure stuff out if we go back to that figure stuff out if we go back to that idea of simulation and this is a idea of simulation and this is a idea of simulation and this is a entertainment kind of giving meaning to entertainment kind of giving meaning to entertainment kind of giving meaning to us the question is how much suffering is us the question is how much suffering is us the question is how much suffering is reasonable for a video game so yeah I

reasonable for a video game so yeah I reasonable for a video game so yeah I don't mind you know a video game where I don't mind you know a video game where I don't mind you know a video game where I get heptic feedback there is a little get heptic feedback there is a little get heptic feedback there is a little bit of shaking maybe I'm a little scared bit of shaking maybe I'm a little scared bit of shaking maybe I'm a little scared I don't want a game where like kids are I don't want a game where like kids are I don't want a game where like kids are tortured tortured tortured literally that seems unethical at least literally that seems unethical at least literally that seems unethical at least by our human standards are you by our human standards are you by our human standards are you suggesting it's possible to remove suggesting it's possible to remove suggesting it's possible to remove suffering if we're looking at human suffering if we're looking at human suffering if we're looking at human civilization as an optimization problem civilization as an optimization problem civilization as an optimization problem so we know there are some humans who so we know there are some humans who so we know there are some humans who because of a mutation don't experience because of a mutation don't experience because of a mutation don't experience physical pain so at least physical pain physical pain so at least physical pain physical pain so at least physical pain can can can be mutated out re-engineered out be mutated out re-engineered out be mutated out re-engineered out suffering in terms of meaning like you suffering in terms of meaning like you suffering in terms of meaning like you burn the only copy of my book is a burn the only copy of my book is a burn the only copy of my book is a little harder but even there you can little harder but even there you can little harder but even there you can manipulate your honic set point you can manipulate your honic set point you can manipulate your honic set point you can change defaults you can reset problem change defaults you can reset problem change defaults you can reset problem with that is if you start messing with with that is if you start messing with with that is if you start messing with your reward Channel you start your reward Channel you start your reward Channel you start wireheading and uh end up bissing out uh wireheading and uh end up bissing out uh wireheading and uh end up bissing out uh a little too much well that's the the a little too much well that's the the a little too much well that's the the question would you really want to live question would you really want to live question would you really want to live in a world where there's no suffering in a world where there's no suffering in a world where there's no suffering that's a dark question is there some that's a dark question is there some that's a dark question is there some level of level of level of suffering that reminds us of what this suffering that reminds us of what this suffering that reminds us of what this is all for is all for is all for I I think we need that but I would I I think we need that but I would I I think we need that but I would change the overall range so right now change the overall range so right now change the overall range so right now it's negative Infinity to kind of it's negative Infinity to kind of it's negative Infinity to kind of positive Infinity pain pleasure AIS I positive Infinity pain pleasure AIS I positive Infinity pain pleasure AIS I would make it like zero to positive would make it like zero to positive would make it like zero to positive infinity and being unhappy is like I'm infinity and being unhappy is like I'm infinity and being unhappy is like I'm close to close to close to zero okay so what what's the srisk what zero okay so what what's the srisk what zero okay so what what's the srisk what are the possible things that you're are the possible things that you're are the possible things that you're imagining with srisk so Mass suffering

imagining with srisk so Mass suffering imagining with srisk so Mass suffering of humans what are we talking about of humans what are we talking about of humans what are we talking about there caused by AGI so there are many there caused by AGI so there are many there caused by AGI so there are many malevolent actors we can talk about malevolent actors we can talk about malevolent actors we can talk about Psychopaths crazies hackers Doomsday Psychopaths crazies hackers Doomsday Psychopaths crazies hackers Doomsday cults we know from history they tried cults we know from history they tried cults we know from history they tried killing everyone they tried on purpose killing everyone they tried on purpose killing everyone they tried on purpose to cause maximum amount of damage to cause maximum amount of damage to cause maximum amount of damage terrorism what if someone malevolent terrorism what if someone malevolent terrorism what if someone malevolent wants on purpose to torture all humans wants on purpose to torture all humans wants on purpose to torture all humans as long as possible you solve aging so as long as possible you solve aging so as long as possible you solve aging so now you have functional immortality and now you have functional immortality and now you have functional immortality and you just try to be as creative as you you just try to be as creative as you you just try to be as creative as you can do you think there is actually can do you think there is actually can do you think there is actually people in human history that try to people in human history that try to people in human history that try to literally maximize human suffering in literally maximize human suffering in literally maximize human suffering in just studying people have done evil in just studying people have done evil in just studying people have done evil in the world it seems that they think that the world it seems that they think that the world it seems that they think that they're doing good and it doesn't seem they're doing good and it doesn't seem they're doing good and it doesn't seem like they're trying to maximize like they're trying to maximize like they're trying to maximize suffering they just cause a lot of suffering they just cause a lot of suffering they just cause a lot of suffering as a side effect of doing what suffering as a side effect of doing what suffering as a side effect of doing what they think is good so there are they think is good so there are they think is good so there are different malevolent agents some may be different malevolent agents some may be different malevolent agents some may be just gaining personal benefit and just gaining personal benefit and just gaining personal benefit and sacrificing others to that cause others sacrificing others to that cause others sacrificing others to that cause others will know for a fact trying to kill as will know for a fact trying to kill as will know for a fact trying to kill as many people as possible when we look at many people as possible when we look at many people as possible when we look at recent school shootings if they had more recent school shootings if they had more recent school shootings if they had more capable weapons they would take out not capable weapons they would take out not capable weapons they would take out not dozens but thousands millions billions well we don't know billions well we don't know that but that is a terrifying that but that is a terrifying that but that is a terrifying possibility and we don't want to find possibility and we don't want to find possibility and we don't want to find out like if terrorists had access to

out like if terrorists had access to out like if terrorists had access to nuclear weapons how far would they go is nuclear weapons how far would they go is nuclear weapons how far would they go is there a limit to what they're there a limit to what they're there a limit to what they're willing to willing to willing to do in your senses there's some do in your senses there's some do in your senses there's some malevolent actors where there's no limit malevolent actors where there's no limit malevolent actors where there's no limit there is there is there is mental mental diseases where people mental mental diseases where people mental mental diseases where people don't have empathy don't don't have empathy don't don't have empathy don't have this have this have this human quality of understanding suffering human quality of understanding suffering human quality of understanding suffering in others and then there's also set of in others and then there's also set of in others and then there's also set of beliefs where you think you're doing beliefs where you think you're doing beliefs where you think you're doing good uh by killing a lot of good uh by killing a lot of good uh by killing a lot of humans again I would like to assume that humans again I would like to assume that humans again I would like to assume that normal people never think like that it's normal people never think like that it's normal people never think like that it's always some sort of psychopaths but yeah always some sort of psychopaths but yeah always some sort of psychopaths but yeah and to you AGI systems can carry that and to you AGI systems can carry that and to you AGI systems can carry that and uh be more competent at executing and uh be more competent at executing and uh be more competent at executing that they can certainly be more creative that they can certainly be more creative that they can certainly be more creative they can understand human biology better they can understand human biology better they can understand human biology better understand our molecular structure understand our molecular structure understand our molecular structure genome uh again uh a lot of times uh genome uh again uh a lot of times uh genome uh again uh a lot of times uh torture ends then in individual dies torture ends then in individual dies torture ends then in individual dies that limit can be removed as well so if that limit can be removed as well so if that limit can be removed as well so if we're actually looking at x risk and we're actually looking at x risk and we're actually looking at x risk and srisk as the systems get more and more srisk as the systems get more and more srisk as the systems get more and more intelligent don't you think it it's intelligent don't you think it it's intelligent don't you think it it's possible to anticipate the ways they can possible to anticipate the ways they can possible to anticipate the ways they can do it and defend against it like we do do it and defend against it like we do do it and defend against it like we do with the cyber security with the do with the cyber security with the do with the cyber security with the do security systems right uh we can security systems right uh we can security systems right uh we can definitely keep up for a while I'm definitely keep up for a while I'm definitely keep up for a while I'm saying you cannot do it indefinitely at saying you cannot do it indefinitely at saying you cannot do it indefinitely at some point the cognitive Gap is too big some point the cognitive Gap is too big some point the cognitive Gap is too big the surface you have to defend is

the surface you have to defend is the surface you have to defend is infinite but attackers only need to find infinite but attackers only need to find infinite but attackers only need to find one exploit so to you eventually this is one exploit so to you eventually this is one exploit so to you eventually this is we're heading off a cliff if we create we're heading off a cliff if we create we're heading off a cliff if we create General super intelligences I don't see General super intelligences I don't see General super intelligences I don't see a good outcome long term for Humanity a good outcome long term for Humanity a good outcome long term for Humanity the only way to win this game is not to the only way to win this game is not to the only way to win this game is not to play it okay well we we we'll talk about play it okay well we we we'll talk about play it okay well we we we'll talk about possible solutions and what not playing possible solutions and what not playing possible solutions and what not playing it means um but what are the possible it means um but what are the possible it means um but what are the possible timelines here to you what are we timelines here to you what are we timelines here to you what are we talking about we're talking about a set talking about we're talking about a set talking about we're talking about a set of years decades centuries what do you of years decades centuries what do you of years decades centuries what do you think I don't know for sure the think I don't know for sure the think I don't know for sure the prediction markets right now are saying prediction markets right now are saying prediction markets right now are saying 2026 for AGI I heard the same thing from 2026 for AGI I heard the same thing from 2026 for AGI I heard the same thing from CEO of anthropic dip mine so maybe we CEO of anthropic dip mine so maybe we CEO of anthropic dip mine so maybe we are 2 years away which seems very are 2 years away which seems very are 2 years away which seems very soon uh given we don't have a working soon uh given we don't have a working soon uh given we don't have a working safety mechanism in place or even a safety mechanism in place or even a safety mechanism in place or even a prototype for one and there are people prototype for one and there are people prototype for one and there are people trying to accelerate those timelines trying to accelerate those timelines trying to accelerate those timelines because they feel we're not getting because they feel we're not getting because they feel we're not getting there quick enough but what do you think there quick enough but what do you think there quick enough but what do you think they mean when they say AGI so the they mean when they say AGI so the they mean when they say AGI so the definitions we used to have when people definitions we used to have when people definitions we used to have when people are modifying a little bit lately are modifying a little bit lately are modifying a little bit lately artificial general intelligence was a artificial general intelligence was a artificial general intelligence was a system capable of performing in any system capable of performing in any system capable of performing in any domain a human could perform so kind of domain a human could perform so kind of domain a human could perform so kind of you creating this average artificial you creating this average artificial you creating this average artificial person they can do cognitive labor person they can do cognitive labor person they can do cognitive labor physical labor where you can get another physical labor where you can get another physical labor where you can get another human to do it superintelligence was human to do it superintelligence was human to do it superintelligence was defined as a system which is superior to defined as a system which is superior to defined as a system which is superior to All Humans in all domains now people are All Humans in all domains now people are All Humans in all domains now people are starting to refer to AGI as if it's starting to refer to AGI as if it's starting to refer to AGI as if it's super intelligence I made a post super intelligence I made a post super intelligence I made a post recently where I argued for me at least

recently where I argued for me at least recently where I argued for me at least if you average out over all the common if you average out over all the common if you average out over all the common human tasks those systems are already human tasks those systems are already human tasks those systems are already smarter than an average human mhm so smarter than an average human mhm so smarter than an average human mhm so under that definition we we have it under that definition we we have it under that definition we we have it Shane L has this definition of where Shane L has this definition of where Shane L has this definition of where you're trying to win in all domains you're trying to win in all domains you're trying to win in all domains that's what intelligence is now are they that's what intelligence is now are they that's what intelligence is now are they smarter than Elite individuals in smarter than Elite individuals in smarter than Elite individuals in certain domains of course not they're certain domains of course not they're certain domains of course not they're not there yet but uh the progress is not there yet but uh the progress is not there yet but uh the progress is exponential see I'm much more concerned exponential see I'm much more concerned exponential see I'm much more concerned about social about social about social engineering so engineering so engineering so to me ai's ability to do something in to me ai's ability to do something in to me ai's ability to do something in the physical world like the the lowest the physical world like the the lowest the physical world like the the lowest hanging fruit this the easiest set of hanging fruit this the easiest set of hanging fruit this the easiest set of methods is by just getting humans to do methods is by just getting humans to do methods is by just getting humans to do it it's going to be much harder to to uh it it's going to be much harder to to uh it it's going to be much harder to to uh be the kind of viruses that take over be the kind of viruses that take over be the kind of viruses that take over the minds of the minds of the minds of robots that where the robots are robots that where the robots are robots that where the robots are executing the commands it just seems executing the commands it just seems executing the commands it just seems like humans social engineering of humans like humans social engineering of humans like humans social engineering of humans is much more likely that would be enough is much more likely that would be enough is much more likely that would be enough to bootst the whole to bootst the whole to bootst the whole process okay just to linger on the term process okay just to linger on the term process okay just to linger on the term AGI what's what to you is the difference AGI what's what to you is the difference AGI what's what to you is the difference between AGI and human level intelligence between AGI and human level intelligence between AGI and human level intelligence uh human level is General in the domain uh human level is General in the domain uh human level is General in the domain of expertise of humans we know how to do of expertise of humans we know how to do of expertise of humans we know how to do human things I don't speak dog language human things I don't speak dog language human things I don't speak dog language I should be able to pick it up if I'm a I should be able to pick it up if I'm a I should be able to pick it up if I'm a general intelligence it's kind of general intelligence it's kind of general intelligence it's kind of inferior animal I should be able to inferior animal I should be able to inferior animal I should be able to learn that skill but I can't at general learn that skill but I can't at general learn that skill but I can't at general intelligence truly Universal general intelligence truly Universal general intelligence truly Universal general intelligence should be able to do things

intelligence should be able to do things intelligence should be able to do things like that humans cannot do to be able to like that humans cannot do to be able to like that humans cannot do to be able to talk to animals for example to solve talk to animals for example to solve talk to animals for example to solve pattern recognition problems of that pattern recognition problems of that pattern recognition problems of that type to type to type to do of similar things outside of do of similar things outside of do of similar things outside of our domain of expertise because it's our domain of expertise because it's our domain of expertise because it's just not the world just not the world just not the world will if we just look at the space of will if we just look at the space of will if we just look at the space of cognitive abilities we have I just would cognitive abilities we have I just would cognitive abilities we have I just would love to understand what the limits are love to understand what the limits are love to understand what the limits are Beyond which an AGI system can reach Beyond which an AGI system can reach Beyond which an AGI system can reach like what does that look like what about like what does that look like what about like what does that look like what about about about about actual mathematical thinking or uh actual mathematical thinking or uh actual mathematical thinking or uh scientific scientific scientific innovation that kind of stuff we know innovation that kind of stuff we know innovation that kind of stuff we know calculators are smarter than humans in calculators are smarter than humans in calculators are smarter than humans in that narrow domain of addition but is it that narrow domain of addition but is it that narrow domain of addition but is it humans plus tools versus AGI or just humans plus tools versus AGI or just humans plus tools versus AGI or just human raw human intelligence cu cu human raw human intelligence cu cu human raw human intelligence cu cu humans create tools and with the tools humans create tools and with the tools humans create tools and with the tools they become more intelligent so like they become more intelligent so like they become more intelligent so like there there's a gray area there what it there there's a gray area there what it there there's a gray area there what it means to be human when we're measuring means to be human when we're measuring means to be human when we're measuring their intelligence so when I think about their intelligence so when I think about their intelligence so when I think about it I usually think human with like a it I usually think human with like a it I usually think human with like a paper and a pencil not human with paper and a pencil not human with paper and a pencil not human with internet and anava AI helping but is internet and anava AI helping but is internet and anava AI helping but is that a fair way to think about it cuz that a fair way to think about it cuz that a fair way to think about it cuz isn't there another definition of human isn't there another definition of human isn't there another definition of human level intelligence that includes the level intelligence that includes the level intelligence that includes the tools that humans create but we create tools that humans create but we create tools that humans create but we create AI so at any point you'll still just add AI so at any point you'll still just add AI so at any point you'll still just add super intelligence to human capability super intelligence to human capability super intelligence to human capability that seems like that seems like that seems like cheating no controllable tools there is cheating no controllable tools there is cheating no controllable tools there is there is an implied leap that you're there is an implied leap that you're there is an implied leap that you're making when AGI goes from tool to uh

making when AGI goes from tool to uh making when AGI goes from tool to uh entity that can make its own decisions entity that can make its own decisions entity that can make its own decisions so if we Define human level intelligence so if we Define human level intelligence so if we Define human level intelligence as everything a human can do with fully as everything a human can do with fully as everything a human can do with fully controllable tools it seems like a controllable tools it seems like a controllable tools it seems like a hybrid of some kind you're now doing hybrid of some kind you're now doing hybrid of some kind you're now doing brain computer interfaces you connecting brain computer interfaces you connecting brain computer interfaces you connecting it to maybe narrow AI yeah it definitely it to maybe narrow AI yeah it definitely it to maybe narrow AI yeah it definitely increases our increases our increases our capabilities so what's a good test to capabilities so what's a good test to capabilities so what's a good test to you that uh measures whether uh an you that uh measures whether uh an you that uh measures whether uh an artificial intelligence system has artificial intelligence system has artificial intelligence system has reached human level intelligence and was reached human level intelligence and was reached human level intelligence and was a good test where it has superseded a good test where it has superseded a good test where it has superseded human level intelligence to reach that human level intelligence to reach that human level intelligence to reach that land of AGI I am oldfashioned I like land of AGI I am oldfashioned I like land of AGI I am oldfashioned I like tting test I have a paper where I equate tting test I have a paper where I equate tting test I have a paper where I equate passing touring test to solving AI passing touring test to solving AI passing touring test to solving AI complete problems because you can encode complete problems because you can encode complete problems because you can encode any questions about any domain into the any questions about any domain into the any questions about any domain into the touring test you don't have to talk touring test you don't have to talk touring test you don't have to talk about how was your day you can ask about how was your day you can ask about how was your day you can ask anything and so the system has to be as anything and so the system has to be as anything and so the system has to be as smart as a human to pass it in a true smart as a human to pass it in a true smart as a human to pass it in a true sense but then you would extend that to sense but then you would extend that to sense but then you would extend that to U maybe a very long conversation like I U maybe a very long conversation like I U maybe a very long conversation like I think the Alexa prize was doing that think the Alexa prize was doing that think the Alexa prize was doing that basically can you do a 20 minute 30 basically can you do a 20 minute 30 basically can you do a 20 minute 30 minute conversation with an ass system minute conversation with an ass system minute conversation with an ass system it has to be long enough to where you it has to be long enough to where you it has to be long enough to where you can make some meaningful decisions about can make some meaningful decisions about can make some meaningful decisions about capabilities absolutely you can Brute capabilities absolutely you can Brute capabilities absolutely you can Brute Force very short Force very short Force very short conversations so like literally what conversations so like literally what conversations so like literally what does that look like can we do uh can we does that look like can we do uh can we does that look like can we do uh can we construct formally a kind of test that

construct formally a kind of test that construct formally a kind of test that tests for AGI for AGI it has to be there tests for AGI for AGI it has to be there tests for AGI for AGI it has to be there I cannot give it a task I can give to a I cannot give it a task I can give to a I cannot give it a task I can give to a human and it cannot do it if a human can human and it cannot do it if a human can human and it cannot do it if a human can for super intelligent it would be for super intelligent it would be for super intelligent it would be superior on all such tasks not just superior on all such tasks not just superior on all such tasks not just average performance so like go learn to average performance so like go learn to average performance so like go learn to drive car go speak Chinese play guitar drive car go speak Chinese play guitar drive car go speak Chinese play guitar okay great I guess the the following okay great I guess the the following okay great I guess the the following question is there a test for the kind of question is there a test for the kind of question is there a test for the kind of AGI that AGI that AGI that would would would be uh susceptible to lead to srisk or X be uh susceptible to lead to srisk or X be uh susceptible to lead to srisk or X risk susceptible to destroy human risk susceptible to destroy human risk susceptible to destroy human civilization like is there a test for civilization like is there a test for civilization like is there a test for that you can develop a test which will that you can develop a test which will that you can develop a test which will give you positives if it lies to you or give you positives if it lies to you or give you positives if it lies to you or has those ideas you cannot develop a has those ideas you cannot develop a has those ideas you cannot develop a test which rules them out there is test which rules them out there is test which rules them out there is always possibility of what bom calls a always possibility of what bom calls a always possibility of what bom calls a treacherous turn where later on a system treacherous turn where later on a system treacherous turn where later on a system decides for game theoretic reasons decides for game theoretic reasons decides for game theoretic reasons economic reasons to change its behavior economic reasons to change its behavior economic reasons to change its behavior and we see the same with humans it's not and we see the same with humans it's not and we see the same with humans it's not unique to AI for Millennia we tried unique to AI for Millennia we tried unique to AI for Millennia we tried developing morals ethics religions uh developing morals ethics religions uh developing morals ethics religions uh light detector tests and then employees light detector tests and then employees light detector tests and then employees betray the employer spouses betray betray the employer spouses betray betray the employer spouses betray family it's a pretty standard thing family it's a pretty standard thing family it's a pretty standard thing intelligent agents sometimes do so is it intelligent agents sometimes do so is it intelligent agents sometimes do so is it is it possible to detect when a AI is it possible to detect when a AI is it possible to detect when a AI system is lying or deceiving you if you system is lying or deceiving you if you system is lying or deceiving you if you know the truth and it tells you know the truth and it tells you know the truth and it tells you something false you can detect that but something false you can detect that but something false you can detect that but you cannot know in general every single

you cannot know in general every single you cannot know in general every single time and again the system you're testing time and again the system you're testing time and again the system you're testing today may not be lying the system you're today may not be lying the system you're today may not be lying the system you're testing today may know you are testing testing today may know you are testing testing today may know you are testing it and so behaving and later on after it it and so behaving and later on after it it and so behaving and later on after it interacts with the environment interacts interacts with the environment interacts interacts with the environment interacts with other systems malevolent agents with other systems malevolent agents with other systems malevolent agents learns more it may start doing those learns more it may start doing those learns more it may start doing those things so do you think it's possible to things so do you think it's possible to things so do you think it's possible to develop a system where the creators of develop a system where the creators of develop a system where the creators of the system the developers the program the system the developers the program the system the developers the program rers don't know that it's deceiving them rers don't know that it's deceiving them rers don't know that it's deceiving them so systems today don't have long-term so systems today don't have long-term so systems today don't have long-term planning that is not out they can lie planning that is not out they can lie planning that is not out they can lie today if it today if it today if it optimizes helps them optimize the reward optimizes helps them optimize the reward optimizes helps them optimize the reward if they realize okay this human will be if they realize okay this human will be if they realize okay this human will be very happy if I tell them the following very happy if I tell them the following very happy if I tell them the following they will do it if it brings them more they will do it if it brings them more they will do it if it brings them more points and they don't have to kind of points and they don't have to kind of points and they don't have to kind of keep track of it it's just the right keep track of it it's just the right keep track of it it's just the right answer to this problem every single time answer to this problem every single time answer to this problem every single time at which point is somebody creating that at which point is somebody creating that at which point is somebody creating that intentionally not unintentionally intentionally not unintentionally intentionally not unintentionally intentionally creating an AI system intentionally creating an AI system intentionally creating an AI system that's doing long-term planning with an that's doing long-term planning with an that's doing long-term planning with an objective function that's defined by the objective function that's defined by the objective function that's defined by the AI system not by a human well some AI system not by a human well some AI system not by a human well some people think that if they're that smart people think that if they're that smart people think that if they're that smart they always good they really do believe they always good they really do believe they always good they really do believe that it's just benevolence from that it's just benevolence from that it's just benevolence from intelligence so they'll always want intelligence so they'll always want intelligence so they'll always want what's best for us some people think what's best for us some people think what's best for us some people think that uh they will be able to detect that uh they will be able to detect that uh they will be able to detect problem behaviors and correct them at problem behaviors and correct them at problem behaviors and correct them at the time when we get

the time when we get the time when we get there I don't think it's a good idea I there I don't think it's a good idea I there I don't think it's a good idea I am strongly against it but yeah there am strongly against it but yeah there am strongly against it but yeah there are quite a few people who in general are quite a few people who in general are quite a few people who in general are so optimistic about this technology are so optimistic about this technology are so optimistic about this technology it could do no wrong they want it it could do no wrong they want it it could do no wrong they want it developed as soon as possible as capable developed as soon as possible as capable developed as soon as possible as capable as possible so there's going to be as possible so there's going to be as possible so there's going to be people who believe the more intelligent people who believe the more intelligent people who believe the more intelligent it is the more benevolent and so it is the more benevolent and so it is the more benevolent and so therefore it should be the one that therefore it should be the one that therefore it should be the one that defines the objective function that it's defines the objective function that it's defines the objective function that it's U optimizing when it's doing long-term U optimizing when it's doing long-term U optimizing when it's doing long-term planning there are even people who say planning there are even people who say planning there are even people who say okay what's so special about humans okay what's so special about humans okay what's so special about humans right we removed the gender bias we're right we removed the gender bias we're right we removed the gender bias we're removing race bias why is this pro-human removing race bias why is this pro-human removing race bias why is this pro-human bias we are polluting the planet we are bias we are polluting the planet we are bias we are polluting the planet we are as you said you know fight a lot of Wars as you said you know fight a lot of Wars as you said you know fight a lot of Wars kind of violent maybe it's better if kind of violent maybe it's better if kind of violent maybe it's better if this super intelligent perfect uh this super intelligent perfect uh this super intelligent perfect uh Society comes and replaces us it's Society comes and replaces us it's Society comes and replaces us it's normal stage in the evolution of our normal stage in the evolution of our normal stage in the evolution of our species yeah so somebody says uh let's species yeah so somebody says uh let's species yeah so somebody says uh let's develop an AI system that removes the develop an AI system that removes the develop an AI system that removes the violent humans from the world and then violent humans from the world and then violent humans from the world and then it turns out that all humans have it turns out that all humans have it turns out that all humans have violence in them or the capacity for violence in them or the capacity for violence in them or the capacity for violence and therefore all humans are violence and therefore all humans are violence and therefore all humans are removed yeah yeah yeah let me ask about removed yeah yeah yeah let me ask about removed yeah yeah yeah let me ask about uh Yan laon he's somebody who uh you've uh Yan laon he's somebody who uh you've uh Yan laon he's somebody who uh you've had a few exchanges had a few exchanges had a few exchanges with and he's somebody who actively with and he's somebody who actively with and he's somebody who actively pushes back against this view that AI is pushes back against this view that AI is pushes back against this view that AI is going to lead to destruction of uh human going to lead to destruction of uh human going to lead to destruction of uh human civilization also known as uh Ai

dorismar and open source are the best dorismar and open source are the best ways to understand and mitigate the ways to understand and mitigate the ways to understand and mitigate the risks and two AI is not something that risks and two AI is not something that risks and two AI is not something that just happens we build it we have agency just happens we build it we have agency just happens we build it we have agency in what it becomes hence we control the in what it becomes hence we control the in what it becomes hence we control the risks we meaning humans it's not some risks we meaning humans it's not some risks we meaning humans it's not some sort of natural phenomena that uh we sort of natural phenomena that uh we sort of natural phenomena that uh we have no control over so can you can you have no control over so can you can you have no control over so can you can you make the case that he's right and can make the case that he's right and can make the case that he's right and can you try to make the case that he's wrong you try to make the case that he's wrong you try to make the case that he's wrong I cannot make a case that he's right I cannot make a case that he's right I cannot make a case that he's right he's wrong in so many ways it's he's wrong in so many ways it's he's wrong in so many ways it's difficult for me to remember all of them difficult for me to remember all of them difficult for me to remember all of them uh he is a Facebook buddy so I have a uh he is a Facebook buddy so I have a uh he is a Facebook buddy so I have a lot of fun uh having those little lot of fun uh having those little lot of fun uh having those little debates with him so I'm trying to debates with him so I'm trying to debates with him so I'm trying to remember the arguments so one he he says remember the arguments so one he he says remember the arguments so one he he says we are not gifted to this intelligence we are not gifted to this intelligence we are not gifted to this intelligence from Aliens we are designing it we are from Aliens we are designing it we are from Aliens we are designing it we are making decisions about it that's not making decisions about it that's not making decisions about it that's not true it was true then we had expert true it was true then we had expert true it was true then we had expert systems symbolic AI decision threes systems symbolic AI decision threes systems symbolic AI decision threes today you set up parameters for a model today you set up parameters for a model today you set up parameters for a model and you water this plant you give it and you water this plant you give it and you water this plant you give it data you give it compute and it grows data you give it compute and it grows data you give it compute and it grows and after it's finished growing into and after it's finished growing into and after it's finished growing into this alien plant you start testing it to this alien plant you start testing it to this alien plant you start testing it to find out what capabilities it has and it find out what capabilities it has and it find out what capabilities it has and it takes years to figure out even for takes years to figure out even for takes years to figure out even for existing models if it's Str for 6 months existing models if it's Str for 6 months existing models if it's Str for 6 months it will take you 2 3 years to figure out it will take you 2 3 years to figure out it will take you 2 3 years to figure out basic capabilities of that system we basic capabilities of that system we basic capabilities of that system we still discover new capabilities in still discover new capabilities in still discover new capabilities in systems which are already out there so systems which are already out there so systems which are already out there so that's that's not the case so just to that's that's not the case so just to that's that's not the case so just to linger on that to you the difference linger on that to you the difference linger on that to you the difference there that there is some level of there that there is some level of there that there is some level of emergent intelligence that happens in emergent intelligence that happens in emergent intelligence that happens in our current

our current our current approaches so stuff that we don't approaches so stuff that we don't approaches so stuff that we don't hardcode in absolutely that's what makes hardcode in absolutely that's what makes hardcode in absolutely that's what makes it so successful then we had to it so successful then we had to it so successful then we had to painstakingly hardcode in everything painstakingly hardcode in everything painstakingly hardcode in everything we didn't have much progress now just we didn't have much progress now just we didn't have much progress now just spend more money and more compute and spend more money and more compute and spend more money and more compute and it's a lot more capable and then the it's a lot more capable and then the it's a lot more capable and then the question is when there is emergent question is when there is emergent question is when there is emergent intelligent intelligent intelligent phenomena what is the ceiling of that phenomena what is the ceiling of that phenomena what is the ceiling of that for you there's no ceiling for uh for for you there's no ceiling for uh for for you there's no ceiling for uh for Yan laon I think there's a kind of Yan laon I think there's a kind of Yan laon I think there's a kind of ceiling that happens that we have full ceiling that happens that we have full ceiling that happens that we have full control over even if we don't understand control over even if we don't understand control over even if we don't understand the internals of the emergence how the the internals of the emergence how the the internals of the emergence how the emergence happens there's a sense that emergence happens there's a sense that emergence happens there's a sense that we have control and understanding of the we have control and understanding of the we have control and understanding of the approximate ceiling of capability the approximate ceiling of capability the approximate ceiling of capability the limits of the capability let's say there limits of the capability let's say there limits of the capability let's say there is a ceiling it's not guaranteed to be is a ceiling it's not guaranteed to be is a ceiling it's not guaranteed to be at a level which is competitive with us at a level which is competitive with us at a level which is competitive with us it may be greatly Superior to ours so it may be greatly Superior to ours so it may be greatly Superior to ours so what what what about his statement about open research about his statement about open research about his statement about open research and open source are the best ways to and open source are the best ways to and open source are the best ways to understand and mitigate the risks understand and mitigate the risks understand and mitigate the risks historically he's completely right open historically he's completely right open historically he's completely right open source software is wonderful it's tested source software is wonderful it's tested source software is wonderful it's tested by the community it's de by the community it's de by the community it's de but we're switching from tools to agents but we're switching from tools to agents but we're switching from tools to agents now you're giving open source weapons to now you're giving open source weapons to now you're giving open source weapons to Psychopaths do we want to open source Psychopaths do we want to open source Psychopaths do we want to open source nuclear weapons biological weapons it's nuclear weapons biological weapons it's nuclear weapons biological weapons it's not safe to give technology so powerful not safe to give technology so powerful not safe to give technology so powerful to those who may misalign it even if you to those who may misalign it even if you to those who may misalign it even if you are successful at somehow getting it to

are successful at somehow getting it to are successful at somehow getting it to work in the first place in a friendly work in the first place in a friendly work in the first place in a friendly manner but the difference with nuclear manner but the difference with nuclear manner but the difference with nuclear weapons current AI systems are not akin weapons current AI systems are not akin weapons current AI systems are not akin to nuclear weapons so the idea there is to nuclear weapons so the idea there is to nuclear weapons so the idea there is you're open sourcing it at this stage you're open sourcing it at this stage you're open sourcing it at this stage that you can understand it better large that you can understand it better large that you can understand it better large large number of people can explore the large number of people can explore the large number of people can explore the limitation the capabilities explore the limitation the capabilities explore the limitation the capabilities explore the possible ways to keep it safe to keep uh possible ways to keep it safe to keep uh possible ways to keep it safe to keep uh it secure all that kind of stuff while it secure all that kind of stuff while it secure all that kind of stuff while it's not at the stage of nuclear weapons it's not at the stage of nuclear weapons it's not at the stage of nuclear weapons so nuclear weapons there's a no nuclear so nuclear weapons there's a no nuclear so nuclear weapons there's a no nuclear weapon and then there's a nuclear weapon weapon and then there's a nuclear weapon weapon and then there's a nuclear weapon with AI systems there's a gradual with AI systems there's a gradual with AI systems there's a gradual Improvement of capability and you get Improvement of capability and you get Improvement of capability and you get to uh perform that Improvement to uh perform that Improvement to uh perform that Improvement incrementally and so open source allows incrementally and so open source allows incrementally and so open source allows you to study you to study you to study uh how things go wrong I study the the uh how things go wrong I study the the uh how things go wrong I study the the very process of emergence study AI very process of emergence study AI very process of emergence study AI safety on those systems when there's not safety on those systems when there's not safety on those systems when there's not a high level of danger all that kind of a high level of danger all that kind of a high level of danger all that kind of stuff it also sets a very wrong stuff it also sets a very wrong stuff it also sets a very wrong precedence so we open sourced model one precedence so we open sourced model one precedence so we open sourced model one model two model three nothing ever bad model two model three nothing ever bad model two model three nothing ever bad happened so obviously we're going to do happened so obviously we're going to do happened so obviously we're going to do it with model four it's just gradual it with model four it's just gradual it with model four it's just gradual Improvement I I don't think it always Improvement I I don't think it always Improvement I I don't think it always works with the precedent like you're not works with the precedent like you're not works with the precedent like you're not stuck doing it the stuck doing it the stuck doing it the way you always did it just uh it's way you always did it just uh it's way you always did it just uh it's that's a precedent of open research and that's a precedent of open research and that's a precedent of open research and open development such that we get to open development such that we get to open development such that we get to learn together and then the first time learn together and then the first time learn together and then the first time there's a sign of there's a sign of there's a sign of danger some dramatic thing happen not a danger some dramatic thing happen not a danger some dramatic thing happen not a thing that destroys human civilization thing that destroys human civilization thing that destroys human civilization but some dramatic demonstration of but some dramatic demonstration of but some dramatic demonstration of capability that can legitimately lead to

capability that can legitimately lead to capability that can legitimately lead to a lot of damage then everybody wakes up a lot of damage then everybody wakes up a lot of damage then everybody wakes up and says okay we need to regulate this and says okay we need to regulate this and says okay we need to regulate this we need to come up with safety mechanism we need to come up with safety mechanism we need to come up with safety mechanism that stops this right but at this time that stops this right but at this time that stops this right but at this time maybe can educate me but I haven't seen maybe can educate me but I haven't seen maybe can educate me but I haven't seen any illustration of significant damage any illustration of significant damage any illustration of significant damage done by intelligent AI systems so I have done by intelligent AI systems so I have done by intelligent AI systems so I have a paper which collects accidents through a paper which collects accidents through a paper which collects accidents through history of AI and they always are history of AI and they always are history of AI and they always are proportionate to capabilities of that proportionate to capabilities of that proportionate to capabilities of that system so if you have Tic Tac to playing system so if you have Tic Tac to playing system so if you have Tic Tac to playing AI it will fail to properly play and AI it will fail to properly play and AI it will fail to properly play and lose the game which it should draw lose the game which it should draw lose the game which it should draw trivial your spell checker will be trivial your spell checker will be trivial your spell checker will be spellward so on uh I stopped collecting spellward so on uh I stopped collecting spellward so on uh I stopped collecting those because there are just too many those because there are just too many those because there are just too many examples of AI failing at what they are examples of AI failing at what they are examples of AI failing at what they are capable of we haven't had terrible capable of we haven't had terrible capable of we haven't had terrible accidents in a sense of billion people accidents in a sense of billion people accidents in a sense of billion people got killed absolutely true but in got killed absolutely true but in got killed absolutely true but in another paper I argue that those another paper I argue that those another paper I argue that those accidents do not actually prevent people accidents do not actually prevent people accidents do not actually prevent people from continuing with research and from continuing with research and from continuing with research and actually they kind of serve like actually they kind of serve like actually they kind of serve like vaccines a vaccine makes your body a vaccines a vaccine makes your body a vaccines a vaccine makes your body a little bit sick so you can handle the little bit sick so you can handle the little bit sick so you can handle the big disease later much better it's the big disease later much better it's the big disease later much better it's the same here people will point out you know same here people will point out you know same here people will point out you know that accident AI accident we had where that accident AI accident we had where that accident AI accident we had where 12 people died everyone's still here 12 12 people died everyone's still here 12 12 people died everyone's still here 12 people is less than smoking kills it's people is less than smoking kills it's people is less than smoking kills it's not a big deal so we continue so in a not a big deal so we continue so in a not a big deal so we continue so in a way it will actually be kind of way it will actually be kind of way it will actually be kind of confirming that it's not that bad it confirming that it's not that bad it confirming that it's not that bad it matters how the deaths happen whether matters how the deaths happen whether matters how the deaths happen whether it's literally Murder By thei system

it's literally Murder By thei system it's literally Murder By thei system then one is a problem but if it's then one is a problem but if it's then one is a problem but if it's accidents because of increased Reliance accidents because of increased Reliance accidents because of increased Reliance on automation for example so when uh on automation for example so when uh on automation for example so when uh airplanes are flying in an automated way airplanes are flying in an automated way airplanes are flying in an automated way maybe the number of plane crashes maybe the number of plane crashes maybe the number of plane crashes increased by 177% or something and then increased by 177% or something and then increased by 177% or something and then you're like okay do we really want to you're like okay do we really want to you're like okay do we really want to rely on automation I think in a case of rely on automation I think in a case of rely on automation I think in a case of automation airplanes it decrease automation airplanes it decrease automation airplanes it decrease significantly okay same thing with significantly okay same thing with significantly okay same thing with autonomous vehicles like okay uh what autonomous vehicles like okay uh what autonomous vehicles like okay uh what are the pros and cons what are the W are the pros and cons what are the W are the pros and cons what are the W with the trade-offs here you can have with the trade-offs here you can have with the trade-offs here you can have that discussion in an honest way but I that discussion in an honest way but I that discussion in an honest way but I think the kind of things we're talking think the kind of things we're talking think the kind of things we're talking about here is mass about here is mass about here is mass scale pain and scale pain and scale pain and suffering caused by AI systems and I suffering caused by AI systems and I suffering caused by AI systems and I think we need to see illustrations of think we need to see illustrations of think we need to see illustrations of that on a very small scale to start to that on a very small scale to start to that on a very small scale to start to understand that this is really damaging understand that this is really damaging understand that this is really damaging versus clippy versus a tool that's versus clippy versus a tool that's versus clippy versus a tool that's really useful to a lot of people to do really useful to a lot of people to do really useful to a lot of people to do learning to do um summarization of text learning to do um summarization of text learning to do um summarization of text to do question answer all that kind of to do question answer all that kind of to do question answer all that kind of stuff to generate videos a tool stuff to generate videos a tool stuff to generate videos a tool fundamentally a tool versus an agent fundamentally a tool versus an agent fundamentally a tool versus an agent that can do a lot a huge amount of that can do a lot a huge amount of that can do a lot a huge amount of damage so you bring up example of cars damage so you bring up example of cars damage so you bring up example of cars yes cars were slowly developed and yes cars were slowly developed and yes cars were slowly developed and integrated if we had no cars and integrated if we had no cars and integrated if we had no cars and somebody came around and said I invented somebody came around and said I invented somebody came around and said I invented this thing it's called cars it's awesome this thing it's called cars it's awesome this thing it's called cars it's awesome it kills like a 100,000 Americans every it kills like a 100,000 Americans every it kills like a 100,000 Americans every year let's deploy it m would we deploy

year let's deploy it m would we deploy year let's deploy it m would we deploy that there's been fear mongering about that there's been fear mongering about that there's been fear mongering about cars for a long time from the the the cars for a long time from the the the cars for a long time from the the the transition from horses cars there's a transition from horses cars there's a transition from horses cars there's a there's a really nice channnel that I there's a really nice channnel that I there's a really nice channnel that I recommend people check out pessimist recommend people check out pessimist recommend people check out pessimist archive that documents all the fear archive that documents all the fear archive that documents all the fear mongering about technology that's mongering about technology that's mongering about technology that's happened throughout history there's happened throughout history there's happened throughout history there's definitely been a lot of fear-mongering definitely been a lot of fear-mongering definitely been a lot of fear-mongering about cars there's a transition period about cars there's a transition period about cars there's a transition period there about cars about how deadly they there about cars about how deadly they there about cars about how deadly they are we can try it took a very long time are we can try it took a very long time are we can try it took a very long time for cars to proliferate to the degree for cars to proliferate to the degree for cars to proliferate to the degree they have now and then you could ask they have now and then you could ask they have now and then you could ask serious questions uh in terms of the serious questions uh in terms of the serious questions uh in terms of the miles traveled the benefit to the miles traveled the benefit to the miles traveled the benefit to the economy the benefit to the quality of economy the benefit to the quality of economy the benefit to the quality of life that cars do versus the number of life that cars do versus the number of life that cars do versus the number of deaths 30 40,000 in the United States deaths 30 40,000 in the United States deaths 30 40,000 in the United States are we willing to pay that price I think are we willing to pay that price I think are we willing to pay that price I think most people when they're rationally most people when they're rationally most people when they're rationally thinking policy makers will say thinking policy makers will say thinking policy makers will say yes it's we want to yes it's we want to yes it's we want to decrease it from 40,000 to zero and do decrease it from 40,000 to zero and do decrease it from 40,000 to zero and do everything we can to decrease it there's everything we can to decrease it there's everything we can to decrease it there's all kinds of policies incentives you can all kinds of policies incentives you can all kinds of policies incentives you can create to decrease the risks uh with the create to decrease the risks uh with the create to decrease the risks uh with the uh deployment of Technology but then you uh deployment of Technology but then you uh deployment of Technology but then you have to weigh the benefits and the risk have to weigh the benefits and the risk have to weigh the benefits and the risk the technology and the same thing would the technology and the same thing would the technology and the same thing would be done with with with AI you need data be done with with with AI you need data be done with with with AI you need data you need to know but if I'm right and you need to know but if I'm right and you need to know but if I'm right and it's unpredictable unexplainable it's unpredictable unexplainable it's unpredictable unexplainable uncontrollable you cannot make this uncontrollable you cannot make this uncontrollable you cannot make this decision we're gaining $10 trillion of decision we're gaining $10 trillion of decision we're gaining $10 trillion of wealth but we're losing we don't know wealth but we're losing we don't know wealth but we're losing we don't know how many people uh you basically have to how many people uh you basically have to how many people uh you basically have to perform an experiment on 8 billion perform an experiment on 8 billion perform an experiment on 8 billion humans without their consent and even if

humans without their consent and even if humans without their consent and even if they want to give you consent they can't they want to give you consent they can't they want to give you consent they can't because they cannot give informed because they cannot give informed because they cannot give informed consent they don't understand those consent they don't understand those consent they don't understand those things things things right that happens when you do when you right that happens when you do when you right that happens when you do when you go from the predictable to the go from the predictable to the go from the predictable to the unpredictable very unpredictable very unpredictable very quickly you just uh but it's not obvious quickly you just uh but it's not obvious quickly you just uh but it's not obvious to me that AI systems would gain to me that AI systems would gain to me that AI systems would gain capability so quickly that you won't be capability so quickly that you won't be capability so quickly that you won't be able to collect enough data to study the able to collect enough data to study the able to collect enough data to study the sa the benefits and sa the benefits and sa the benefits and risks we literally doing it the previous risks we literally doing it the previous risks we literally doing it the previous model we learned about after we finish model we learned about after we finish model we learned about after we finish training it what it was capable of let's training it what it was capable of let's training it what it was capable of let's say we stopped GPT for training run say we stopped GPT for training run say we stopped GPT for training run around human cap capability around human cap capability around human cap capability hypothetically we start training GPT 5 hypothetically we start training GPT 5 hypothetically we start training GPT 5 and I have no knowledge of Insider and I have no knowledge of Insider and I have no knowledge of Insider training runs or anything and we started training runs or anything and we started training runs or anything and we started that point of about human and we train that point of about human and we train that point of about human and we train it for the next 9 months maybe 2 months it for the next 9 months maybe 2 months it for the next 9 months maybe 2 months in it becomes super intelligent we in it becomes super intelligent we in it becomes super intelligent we continue training it at the time when we continue training it at the time when we continue training it at the time when we start uh testing it it is already a start uh testing it it is already a start uh testing it it is already a dangerous system how dangerous I have no dangerous system how dangerous I have no dangerous system how dangerous I have no idea but neither people training it at idea but neither people training it at idea but neither people training it at the training stage but then there's a the training stage but then there's a the training stage but then there's a testing stage mhm inside the company testing stage mhm inside the company testing stage mhm inside the company they can start getting intuition about they can start getting intuition about they can start getting intuition about what the system is capable to do you're what the system is capable to do you're what the system is capable to do you're saying that somehow from leap from GPT 4 saying that somehow from leap from GPT 4 saying that somehow from leap from GPT 4 to GPT 5 can to GPT 5 can to GPT 5 can happen the kind of leap where GPT 4 was happen the kind of leap where GPT 4 was happen the kind of leap where GPT 4 was controllable in GPT 5 is no longer controllable in GPT 5 is no longer controllable in GPT 5 is no longer controllable and we get no insights from controllable and we get no insights from controllable and we get no insights from using GPT 4 about the fact that GPT 5 using GPT 4 about the fact that GPT 5 using GPT 4 about the fact that GPT 5 will be will be will be uncontrollable like that's the that's

uncontrollable like that's the that's uncontrollable like that's the that's the situation you're concerned about the situation you're concerned about the situation you're concerned about where there leap from n to n plus one where there leap from n to n plus one where there leap from n to n plus one would be such that uncontrollable system would be such that uncontrollable system would be such that uncontrollable system is created is created is created without without without any ability for us to anticipate that if any ability for us to anticipate that if any ability for us to anticipate that if we had capability of ahead of the run we had capability of ahead of the run we had capability of ahead of the run before the training run to register before the training run to register before the training run to register exactly what capabilities that next exactly what capabilities that next exactly what capabilities that next model will have at the end of a training model will have at the end of a training model will have at the end of a training run and we accurately guessed all of run and we accurately guessed all of run and we accurately guessed all of them I would say you're right we can them I would say you're right we can them I would say you're right we can definitely go ahead with this run we definitely go ahead with this run we definitely go ahead with this run we don't have that capability from gp4 you don't have that capability from gp4 you don't have that capability from gp4 you can build up intuition about what GPT 5 can build up intuition about what GPT 5 can build up intuition about what GPT 5 will be capable of it's just incremental will be capable of it's just incremental will be capable of it's just incremental progress MH even if that's a big leap in progress MH even if that's a big leap in progress MH even if that's a big leap in capability it just doesn't seem like you capability it just doesn't seem like you capability it just doesn't seem like you can take a leap from a system that's uh can take a leap from a system that's uh can take a leap from a system that's uh helping you write emails to a system helping you write emails to a system helping you write emails to a system that's going to destroy human that's going to destroy human that's going to destroy human civilization it seems like it's always civilization it seems like it's always civilization it seems like it's always going to be sufficiently incremental going to be sufficiently incremental going to be sufficiently incremental such that we can anticipate the possible such that we can anticipate the possible such that we can anticipate the possible dangers and we're not even talking about dangers and we're not even talking about dangers and we're not even talking about existential risks but just the the kind existential risks but just the the kind existential risks but just the the kind of damage can do to civilization it of damage can do to civilization it of damage can do to civilization it seems like we'll be able to anticipate seems like we'll be able to anticipate seems like we'll be able to anticipate the kinds not the exact but the kinds of the kinds not the exact but the kinds of the kinds not the exact but the kinds of uh risks it might lead to and then uh risks it might lead to and then uh risks it might lead to and then rapidly develop defenses ahead of time rapidly develop defenses ahead of time rapidly develop defenses ahead of time and as the risks emerge we're not and as the risks emerge we're not and as the risks emerge we're not talking just about capabilities specific talking just about capabilities specific talking just about capabilities specific tasks we're talking about General tasks we're talking about General tasks we're talking about General capability to learn maybe like a child

capability to learn maybe like a child capability to learn maybe like a child at the time of testing and deployment it at the time of testing and deployment it at the time of testing and deployment it is still not extremely capable but as it is still not extremely capable but as it is still not extremely capable but as it is exposed to more data real world it is exposed to more data real world it is exposed to more data real world it can be trained to become much more can be trained to become much more can be trained to become much more dangerous and capable so let's let's dangerous and capable so let's let's dangerous and capable so let's let's focus then on the control problem at which point does the system problem at which point does the system become become become uncontrollable why is it the more likely uncontrollable why is it the more likely uncontrollable why is it the more likely trajectory for you that the system trajectory for you that the system trajectory for you that the system becomes becomes becomes uncontrollable so I think at some point uncontrollable so I think at some point uncontrollable so I think at some point it becomes capable of getting out of it becomes capable of getting out of it becomes capable of getting out of control for game theoretic reasons it control for game theoretic reasons it control for game theoretic reasons it may decide not to do anything right away may decide not to do anything right away may decide not to do anything right away and for a long time just collect more and for a long time just collect more and for a long time just collect more resources resources resources accumulate strategic Advantage right accumulate strategic Advantage right accumulate strategic Advantage right away it may be kind of still young weak away it may be kind of still young weak away it may be kind of still young weak super intelligence give it a decade it's super intelligence give it a decade it's super intelligence give it a decade it's in charge of a lot more resources it had in charge of a lot more resources it had in charge of a lot more resources it had time to make backups so it's not obvious time to make backups so it's not obvious time to make backups so it's not obvious to me that it will strike as soon as it to me that it will strike as soon as it to me that it will strike as soon as it can can we just try to imagine this can can we just try to imagine this can can we just try to imagine this future with there's an AI system that's future with there's an AI system that's future with there's an AI system that's capable of capable of capable of uh escaping in control of humans and uh escaping in control of humans and uh escaping in control of humans and then doesn't and waits what's that look then doesn't and waits what's that look then doesn't and waits what's that look like so one we have to rely on that like so one we have to rely on that like so one we have to rely on that system for a lot of the infrastructure system for a lot of the infrastructure system for a lot of the infrastructure so we have to give it access not just to so we have to give it access not just to so we have to give it access not just to the internet but to the task of the internet but to the task of the internet but to the task of managing uh Power government economy managing uh Power government economy managing uh Power government economy this kind of stuff so and that just this kind of stuff so and that just this kind of stuff so and that just feels like a gradual process given the feels like a gradual process given the feels like a gradual process given the bureaucracies of all those systems

bureaucracies of all those systems bureaucracies of all those systems involved we've been doing it for years involved we've been doing it for years involved we've been doing it for years software controls all those systems software controls all those systems software controls all those systems nuclear power plants airline industry nuclear power plants airline industry nuclear power plants airline industry it's all software based every time there it's all software based every time there it's all software based every time there is electrical outage I can't fly is electrical outage I can't fly is electrical outage I can't fly anywhere for days but there's a anywhere for days but there's a anywhere for days but there's a difference between difference between difference between software and software and software and AI there's different kinds of software AI there's different kinds of software AI there's different kinds of software so to give a single AI system access to so to give a single AI system access to so to give a single AI system access to the control of Airlines and the control the control of Airlines and the control the control of Airlines and the control of the of the of the economy that's not a that's not a economy that's not a that's not a economy that's not a that's not a trivial transition for Humanity no but trivial transition for Humanity no but trivial transition for Humanity no but if it shows it is safer in fact fact if it shows it is safer in fact fact if it shows it is safer in fact fact then it's in control we get better then it's in control we get better then it's in control we get better results people will demand that it put results people will demand that it put results people will demand that it put in place and if not it can hack the in place and if not it can hack the in place and if not it can hack the system it can use social engineering to system it can use social engineering to system it can use social engineering to get access to it that's why I said it get access to it that's why I said it get access to it that's why I said it might take some time for it to might take some time for it to might take some time for it to accumulate those resources it just feels accumulate those resources it just feels accumulate those resources it just feels like that would take a long time for like that would take a long time for like that would take a long time for either humans to trust it or for the either humans to trust it or for the either humans to trust it or for the social engineering to come into play social engineering to come into play social engineering to come into play like it's not a thing that happens like it's not a thing that happens like it's not a thing that happens overnight it feels like something that overnight it feels like something that overnight it feels like something that happens across one or two decades I happens across one or two decades I happens across one or two decades I really hope you're right but it's not really hope you're right but it's not really hope you're right but it's not what I'm seeing people are very quick to what I'm seeing people are very quick to what I'm seeing people are very quick to jump on a latest Trend early adopters jump on a latest Trend early adopters jump on a latest Trend early adopters will be there before it's even deployed will be there before it's even deployed will be there before it's even deployed buying prototypes maybe the social buying prototypes maybe the social buying prototypes maybe the social engineering I can see because so for engineering I can see because so for engineering I can see because so for social engineering AI systems don't need social engineering AI systems don't need social engineering AI systems don't need any hardware access they just it's all any hardware access they just it's all any hardware access they just it's all software so they can start manipulating software so they can start manipulating software so they can start manipulating you through social media so on like you you through social media so on like you you through social media so on like you have ai assistants they're going to help have ai assistants they're going to help have ai assistants they're going to help you do a lot of manage a lot of your you do a lot of manage a lot of your you do a lot of manage a lot of your day-to-day and then they start doing day-to-day and then they start doing day-to-day and then they start doing social engineering but like for a system

social engineering but like for a system social engineering but like for a system that's so capable that is can escape the that's so capable that is can escape the that's so capable that is can escape the control of humans that created it such a control of humans that created it such a control of humans that created it such a system being deployed at a mass system being deployed at a mass system being deployed at a mass scale and trusted by people to be scale and trusted by people to be scale and trusted by people to be deployed it feels like that would take a deployed it feels like that would take a deployed it feels like that would take a lot of lot of lot of convincing so we've been deploying convincing so we've been deploying convincing so we've been deploying systems which had hidden systems which had hidden systems which had hidden capabilities can you give an example gp4 capabilities can you give an example gp4 capabilities can you give an example gp4 I don't know what else is capable of but I don't know what else is capable of but I don't know what else is capable of but there are still things we haven't there are still things we haven't there are still things we haven't discovered can do there may be trial discovered can do there may be trial discovered can do there may be trial proportional to his capability I don't proportional to his capability I don't proportional to his capability I don't know it writes Chinese poetry know it writes Chinese poetry know it writes Chinese poetry hypothetical I know it does but we hypothetical I know it does but we hypothetical I know it does but we haven't tested for all possible haven't tested for all possible haven't tested for all possible capabilities and we are not explicitly capabilities and we are not explicitly capabilities and we are not explicitly designing them MH we can only rule out designing them MH we can only rule out designing them MH we can only rule out bugs we find we cannot rule out bugs and bugs we find we cannot rule out bugs and bugs we find we cannot rule out bugs and capabilities because we haven't found capabilities because we haven't found capabilities because we haven't found them is it possible for a system to have them is it possible for a system to have them is it possible for a system to have hidden hidden hidden capabilities that are orders a magnitude capabilities that are orders a magnitude capabilities that are orders a magnitude greater than its non-hidden greater than its non-hidden greater than its non-hidden capabilities this is the thing I'm capabilities this is the thing I'm capabilities this is the thing I'm really struggling with where on the really struggling with where on the really struggling with where on the surface the thing we understand it can surface the thing we understand it can surface the thing we understand it can do doesn't seem that harmful so if even do doesn't seem that harmful so if even do doesn't seem that harmful so if even if it has bugs even if it has hidden if it has bugs even if it has hidden if it has bugs even if it has hidden capabilities like Chinese poetry or capabilities like Chinese poetry or capabilities like Chinese poetry or generating effective generating effective generating effective viruses uh software viruses uh software viruses uh software viruses the damage that can do seems viruses the damage that can do seems viruses the damage that can do seems like on the same order of magnitude as

like on the same order of magnitude as like on the same order of magnitude as it's it's it's uh the the capabilities that we know uh the the capabilities that we know uh the the capabilities that we know about so like this this idea that the about so like this this idea that the about so like this this idea that the hidden capabilities will include being hidden capabilities will include being hidden capabilities will include being uncontrollable this is something I'm uncontrollable this is something I'm uncontrollable this is something I'm struggling with cuz GPT 4 on the surface struggling with cuz GPT 4 on the surface struggling with cuz GPT 4 on the surface seems to be very controllable again we seems to be very controllable again we seems to be very controllable again we can only ask and test for things we know can only ask and test for things we know can only ask and test for things we know about if there are unknown unknowns we about if there are unknown unknowns we about if there are unknown unknowns we cannot do it I'm thinking of human cannot do it I'm thinking of human cannot do it I'm thinking of human statistics of an right if you talk to a statistics of an right if you talk to a statistics of an right if you talk to a person like that you may not even person like that you may not even person like that you may not even realize they can multiply 20 digit realize they can multiply 20 digit realize they can multiply 20 digit number numbers in their head you have to number numbers in their head you have to number numbers in their head you have to know to know to know to ask so as I mentioned just to sort of ask so as I mentioned just to sort of ask so as I mentioned just to sort of Linger on Linger on Linger on the the fear of the the the fear of the the the fear of the unknown so the pessimist archive has unknown so the pessimist archive has unknown so the pessimist archive has just documented let's look at data of just documented let's look at data of just documented let's look at data of the past at history there's been a lot the past at history there's been a lot the past at history there's been a lot of fearmongering about technology of fearmongering about technology of fearmongering about technology pessimist archive does a really good job pessimist archive does a really good job pessimist archive does a really good job of documenting how crazily afraid we are of documenting how crazily afraid we are of documenting how crazily afraid we are of every piece of technology we've been of every piece of technology we've been of every piece of technology we've been afraid there's a blog post where anlo afraid there's a blog post where anlo afraid there's a blog post where anlo who created pessimus archive writes who created pessimus archive writes who created pessimus archive writes about the fact that we've been uh about the fact that we've been uh about the fact that we've been uh fear-mongering about robots and fear-mongering about robots and fear-mongering about robots and automation for for over 100 years so why automation for for over 100 years so why automation for for over 100 years so why is Agi different than the kinds of is Agi different than the kinds of is Agi different than the kinds of Technologies we've been afraid of in the Technologies we've been afraid of in the Technologies we've been afraid of in the past so two things one we switching from past so two things one we switching from past so two things one we switching from tools to agents tools don't have tools to agents tools don't have tools to agents tools don't have negative or positive impact people using negative or positive impact people using negative or positive impact people using tools do so guns don't kill people with

tools do so guns don't kill people with tools do so guns don't kill people with guns do agents can make their own guns do agents can make their own guns do agents can make their own decisions they can be positive or decisions they can be positive or decisions they can be positive or negative a pitbull can decide to harm negative a pitbull can decide to harm negative a pitbull can decide to harm you it's an agent the fears are the same you it's an agent the fears are the same you it's an agent the fears are the same the only difference is now we have this the only difference is now we have this the only difference is now we have this technology then they were afraid of technology then they were afraid of technology then they were afraid of humano robots 100 years ago they had humano robots 100 years ago they had humano robots 100 years ago they had none today every major company in the none today every major company in the none today every major company in the world is investing billions to create world is investing billions to create world is investing billions to create them not every but you understand what them not every but you understand what them not every but you understand what I'm saying yes it's very different well I'm saying yes it's very different well I'm saying yes it's very different well agents agents agents uh it depends on what you mean by the uh it depends on what you mean by the uh it depends on what you mean by the word agents the all those companies are word agents the all those companies are word agents the all those companies are not investing in a system that has the not investing in a system that has the not investing in a system that has the kind of kind of kind of agency that's implied by in the fears agency that's implied by in the fears agency that's implied by in the fears where it can really make decisions on where it can really make decisions on where it can really make decisions on their own that have no human in the their own that have no human in the their own that have no human in the loop they are saying they're building loop they are saying they're building loop they are saying they're building super intelligence and have a super super intelligence and have a super super intelligence and have a super alignment team you don't think they're alignment team you don't think they're alignment team you don't think they're trying to create a system smart enough trying to create a system smart enough trying to create a system smart enough to be an independent agent under that to be an independent agent under that to be an independent agent under that definition I have not seen evidence of definition I have not seen evidence of definition I have not seen evidence of it I I think a lot of it is marketing it I I think a lot of it is marketing it I I think a lot of it is marketing uh is is a is a marketing kind of uh is is a is a marketing kind of uh is is a is a marketing kind of discussion about the future and it's a discussion about the future and it's a discussion about the future and it's a it's a mission about the kind of systems it's a mission about the kind of systems it's a mission about the kind of systems we can create in the long-term future we can create in the long-term future we can create in the long-term future but in the short term the kind of but in the short term the kind of but in the short term the kind of systems they're creating Falls fully systems they're creating Falls fully systems they're creating Falls fully within the definition of narrow AI these within the definition of narrow AI these within the definition of narrow AI these are tools that have increasing are tools that have increasing are tools that have increasing capabilities but they're just don't have capabilities but they're just don't have capabilities but they're just don't have a sense of agency or Consciousness or a sense of agency or Consciousness or a sense of agency or Consciousness or self-awareness or ability to deceive at self-awareness or ability to deceive at self-awareness or ability to deceive at Scales that would require would be

Scales that would require would be Scales that would require would be required to do like Mass scale suffering required to do like Mass scale suffering required to do like Mass scale suffering and murder of humans those systems are and murder of humans those systems are and murder of humans those systems are well beyond Naro AI if you had to list well beyond Naro AI if you had to list well beyond Naro AI if you had to list all the capabilities of GPT 4 you would all the capabilities of GPT 4 you would all the capabilities of GPT 4 you would spend a lot of time writing that list spend a lot of time writing that list spend a lot of time writing that list but agency is not one of them not yet but agency is not one of them not yet but agency is not one of them not yet but do you think any of those companies but do you think any of those companies but do you think any of those companies are holding back because they think it are holding back because they think it are holding back because they think it may be not safe or are they developing may be not safe or are they developing may be not safe or are they developing the most capable system they can give the most capable system they can give the most capable system they can give the resources and hoping they can the resources and hoping they can the resources and hoping they can control and control and control and monetize control and monetize hoping monetize control and monetize hoping monetize control and monetize hoping they can control and monetize so you're they can control and monetize so you're they can control and monetize so you're saying if they could press a button and saying if they could press a button and saying if they could press a button and create an create an create an agent that they no longer control that agent that they no longer control that agent that they no longer control that they can have to ask they can have to ask they can have to ask nicely a thing that's lives on a server nicely a thing that's lives on a server nicely a thing that's lives on a server across huge number of uh across huge number of uh across huge number of uh computers you're saying that they would computers you're saying that they would computers you're saying that they would uh push for the creation of that kind of uh push for the creation of that kind of uh push for the creation of that kind of system I mean I can't speak for other system I mean I can't speak for other system I mean I can't speak for other people for all of them I think some of people for all of them I think some of people for all of them I think some of them are very ambitious they fundraising them are very ambitious they fundraising them are very ambitious they fundraising trillions they talk about controlling trillions they talk about controlling trillions they talk about controlling the light corn of the Universe I would the light corn of the Universe I would the light corn of the Universe I would guess that they guess that they guess that they might well that's a human question might well that's a human question might well that's a human question whether humans are capable of that whether humans are capable of that whether humans are capable of that probably some humans are capable of that probably some humans are capable of that probably some humans are capable of that my more direct question if it's possible my more direct question if it's possible my more direct question if it's possible to create such a to create such a to create such a system have a system that has that level system have a system that has that level system have a system that has that level of of of agency I I don't think that's an easy agency I I don't think that's an easy agency I I don't think that's an easy technical technical technical challenge we're not it doesn't I feel challenge we're not it doesn't I feel challenge we're not it doesn't I feel like we're close to that A system that

like we're close to that A system that like we're close to that A system that has the kind of agency where it can make has the kind of agency where it can make has the kind of agency where it can make its own decisions and deceive everybody its own decisions and deceive everybody its own decisions and deceive everybody about them the current about them the current about them the current architecture we have in machine learning architecture we have in machine learning architecture we have in machine learning and how we train the systems how deploy and how we train the systems how deploy and how we train the systems how deploy the systems and all that it just doesn't the systems and all that it just doesn't the systems and all that it just doesn't seem to support that kind of agency I seem to support that kind of agency I seem to support that kind of agency I really hope you are right uh I think the really hope you are right uh I think the really hope you are right uh I think the scaling hypothesis is correct we haven't scaling hypothesis is correct we haven't scaling hypothesis is correct we haven't seen diminishing returns it used to be seen diminishing returns it used to be seen diminishing returns it used to be we asked how long before AGI now we we asked how long before AGI now we we asked how long before AGI now we should ask how much until AGI it's should ask how much until AGI it's should ask how much until AGI it's trillion dollars today it's a billion trillion dollars today it's a billion trillion dollars today it's a billion dollars next year it's a million dollar dollars next year it's a million dollar dollars next year it's a million dollar in a few years don't you think it's in a few years don't you think it's in a few years don't you think it's possible basically run out of possible basically run out of possible basically run out of trillions so is this constrained by trillions so is this constrained by trillions so is this constrained by compute compute gets cheaper every day compute compute gets cheaper every day compute compute gets cheaper every day exponentially but then then that becomes exponentially but then then that becomes exponentially but then then that becomes a question of decades versus years if a question of decades versus years if a question of decades versus years if the only disagreement is that it will the only disagreement is that it will the only disagreement is that it will take decades not years for everything take decades not years for everything take decades not years for everything I'm saying to materialize then I can go I'm saying to materialize then I can go I'm saying to materialize then I can go with that with that with that but if it takes decades then uh the but if it takes decades then uh the but if it takes decades then uh the development of tools for AI development of tools for AI development of tools for AI safety uh becomes more and more safety uh becomes more and more safety uh becomes more and more realistic so I guess the question realistic so I guess the question realistic so I guess the question is I have a fundamental belief that is I have a fundamental belief that is I have a fundamental belief that humans when faced with danger can come humans when faced with danger can come humans when faced with danger can come up with ways to defend defend against up with ways to defend defend against up with ways to defend defend against that that that danger and one of the big problems danger and one of the big problems danger and one of the big problems facing AI safety currently for me is facing AI safety currently for me is facing AI safety currently for me is that there's not clear illustrations of that there's not clear illustrations of that there's not clear illustrations of what that danger looks what that danger looks what that danger looks like there's no illustrations of AI like there's no illustrations of AI like there's no illustrations of AI systems doing a lot of damage and so

systems doing a lot of damage and so systems doing a lot of damage and so it's unclear what you're defending it's unclear what you're defending it's unclear what you're defending against because currently it's a against because currently it's a against because currently it's a philosophical Notions that yes it's philosophical Notions that yes it's philosophical Notions that yes it's possible to imagine AI systems that take possible to imagine AI systems that take possible to imagine AI systems that take control of everything and Destroy All control of everything and Destroy All control of everything and Destroy All Humans it's also a more formal Humans it's also a more formal Humans it's also a more formal mathematical notion that you talk about mathematical notion that you talk about mathematical notion that you talk about that it's impossible to have a perfectly that it's impossible to have a perfectly that it's impossible to have a perfectly secure system you can't you can't prove secure system you can't you can't prove secure system you can't you can't prove that a program of sufficient complexity that a program of sufficient complexity that a program of sufficient complexity is uh completely safe and and perfect is uh completely safe and and perfect is uh completely safe and and perfect and you know everything about it yes but and you know everything about it yes but and you know everything about it yes but like when you actually just like when you actually just like when you actually just pragmatically look how much damage have pragmatically look how much damage have pragmatically look how much damage have the AI systems done and what kind of the AI systems done and what kind of the AI systems done and what kind of damage there's not been illustrations of damage there's not been illustrations of damage there's not been illustrations of that even in autonomous weapon that even in autonomous weapon that even in autonomous weapon systems there's not been mass systems there's not been mass systems there's not been mass deployments of autonomous weapon systems deployments of autonomous weapon systems deployments of autonomous weapon systems luckily um the Automation in war luckily um the Automation in war luckily um the Automation in war currently is very currently is very currently is very limited the that the automation is at limited the that the automation is at limited the that the automation is at the scale of individuals versus like at the scale of individuals versus like at the scale of individuals versus like at the scale of strategy and planning so I the scale of strategy and planning so I the scale of strategy and planning so I think one of the challenges here is like think one of the challenges here is like think one of the challenges here is like where is the where is the where is the dangers uh and the intuition that yam dangers uh and the intuition that yam dangers uh and the intuition that yam Lun and others have is let's keep in the Lun and others have is let's keep in the Lun and others have is let's keep in the open building AI systems until the open building AI systems until the open building AI systems until the dangers start rearing their dangers start rearing their dangers start rearing their heads and they become more explicit heads and they become more explicit heads and they become more explicit there there start being uh case studies there there start being uh case studies there there start being uh case studies illustrative uh case studies that show illustrative uh case studies that show illustrative uh case studies that show exactly how the damage by as systems is

exactly how the damage by as systems is exactly how the damage by as systems is done then regulation can step in then done then regulation can step in then done then regulation can step in then brilliant Engineers can step up and we brilliant Engineers can step up and we brilliant Engineers can step up and we can have Manhattan style projects that can have Manhattan style projects that can have Manhattan style projects that defend against such systems that's kind defend against such systems that's kind defend against such systems that's kind of the no the of the no the of the no the notion and I guess attention with that notion and I guess attention with that notion and I guess attention with that is the idea that for you we need to be is the idea that for you we need to be is the idea that for you we need to be thinking about that now so that we're thinking about that now so that we're thinking about that now so that we're we're ready because we we'll have not we're ready because we we'll have not we're ready because we we'll have not much time once the systems are much time once the systems are much time once the systems are deployed is that true so there is a lot deployed is that true so there is a lot deployed is that true so there is a lot to unpack here uh there is a partnership to unpack here uh there is a partnership to unpack here uh there is a partnership on AI a conglomerate of many large on AI a conglomerate of many large on AI a conglomerate of many large corporations they have a database of AI corporations they have a database of AI corporations they have a database of AI accidents they collect I contributed a accidents they collect I contributed a accidents they collect I contributed a lot to that database if we so far made lot to that database if we so far made lot to that database if we so far made almost no progress in actually solving almost no progress in actually solving almost no progress in actually solving this problem not patching it not again this problem not patching it not again this problem not patching it not again lipstick and a p kind of lipstick and a p kind of lipstick and a p kind of solutions why would we think we'll do solutions why would we think we'll do solutions why would we think we'll do better than we closer to the better than we closer to the better than we closer to the problem uh all the things you mentioned problem uh all the things you mentioned problem uh all the things you mentioned are serious concerns measuring the are serious concerns measuring the are serious concerns measuring the amount of harm so benefit versus risk amount of harm so benefit versus risk amount of harm so benefit versus risk there is is difficult but to you the there is is difficult but to you the there is is difficult but to you the sense is already the risk has superseded sense is already the risk has superseded sense is already the risk has superseded the benefit again I I want to be the benefit again I I want to be the benefit again I I want to be perfectly clear I love AI I love perfectly clear I love AI I love perfectly clear I love AI I love technology I'm a computer scientist I technology I'm a computer scientist I technology I'm a computer scientist I have PhD in engineering I work at an have PhD in engineering I work at an have PhD in engineering I work at an engineering school there is a huge engineering school there is a huge engineering school there is a huge difference between we need to develop difference between we need to develop difference between we need to develop narrow AI systems super intelligent in narrow AI systems super intelligent in narrow AI systems super intelligent in solving specific human problems like solving specific human problems like solving specific human problems like protein folding and let's create super protein folding and let's create super protein folding and let's create super intelligent machine G and will decide intelligent machine G and will decide intelligent machine G and will decide what to do with us yeah those not the what to do with us yeah those not the what to do with us yeah those not the same I am against the super intelligence same I am against the super intelligence same I am against the super intelligence in general sense with No undo button do

in general sense with No undo button do in general sense with No undo button do you think the teams that are doing you think the teams that are doing you think the teams that are doing they're able to do the AI safety on the they're able to do the AI safety on the they're able to do the AI safety on the the kind of narrow AI the kind of narrow AI the kind of narrow AI risks that you've risks that you've risks that you've mentioned are those approaches going to mentioned are those approaches going to mentioned are those approaches going to be at all productive towards leading to be at all productive towards leading to be at all productive towards leading to approaches of doing AI safety on approaches of doing AI safety on approaches of doing AI safety on AGI or is it just a fundamentally AGI or is it just a fundamentally AGI or is it just a fundamentally different partially but they don't scale different partially but they don't scale different partially but they don't scale for narrow AI for deterministic systems for narrow AI for deterministic systems for narrow AI for deterministic systems you can test them you have edge cases you can test them you have edge cases you can test them you have edge cases you know what the answer should look you know what the answer should look you know what the answer should look like you know the right answers for like you know the right answers for like you know the right answers for General systems you have infinite test General systems you have infinite test General systems you have infinite test surface you have no edge cases you surface you have no edge cases you surface you have no edge cases you cannot even know what to test for again cannot even know what to test for again cannot even know what to test for again the unknown unknowns are under the unknown unknowns are under the unknown unknowns are under underappreciated by people looking at underappreciated by people looking at underappreciated by people looking at this problem you are always asking me this problem you are always asking me this problem you are always asking me how will it kill everyone how will it how will it kill everyone how will it how will it kill everyone how will it will fail the whole point is if I knew will fail the whole point is if I knew will fail the whole point is if I knew it would be super intelligent and it would be super intelligent and it would be super intelligent and despite what you might think I'm despite what you might think I'm despite what you might think I'm not so to you the concern is that we not so to you the concern is that we not so to you the concern is that we would not be able would not be able would not be able to see early signs of an uncontrollable to see early signs of an uncontrollable to see early signs of an uncontrollable system it is a master at Deception Sam system it is a master at Deception Sam system it is a master at Deception Sam tweeted about how great it is at tweeted about how great it is at tweeted about how great it is at persuasion and we see it ourselves persuasion and we see it ourselves persuasion and we see it ourselves especially now with voices with maybe especially now with voices with maybe especially now with voices with maybe kind of flirty sarcastic female voices kind of flirty sarcastic female voices kind of flirty sarcastic female voices it's going to be very good at getting it's going to be very good at getting it's going to be very good at getting people to do things but people to do things but people to do things but uh see I'm very uh see I'm very uh see I'm very concerned about system being used to concerned about system being used to concerned about system being used to control the control the control the masses but in that case the developers

masses but in that case the developers masses but in that case the developers know about the kind of control that's know about the kind of control that's know about the kind of control that's happening you're more concerned about happening you're more concerned about happening you're more concerned about the next stage where even the developers the next stage where even the developers the next stage where even the developers don't know about the deception right I don't know about the deception right I don't know about the deception right I don't think developers know everything don't think developers know everything don't think developers know everything about what they are creating they have about what they are creating they have about what they are creating they have lots of great knowledge we're making lots of great knowledge we're making lots of great knowledge we're making progress on explaining parts of a progress on explaining parts of a progress on explaining parts of a network we can understand okay this note network we can understand okay this note network we can understand okay this note get excited then this uh input is get excited then this uh input is get excited then this uh input is presented this cluster of nodes but presented this cluster of nodes but presented this cluster of nodes but we're nowhere near close to we're nowhere near close to we're nowhere near close to understanding the full picture and I understanding the full picture and I understanding the full picture and I think it's impossible you need to be think it's impossible you need to be think it's impossible you need to be able to survey an explanation the size able to survey an explanation the size able to survey an explanation the size of those models prevents a single human of those models prevents a single human of those models prevents a single human from absorbing all this information even from absorbing all this information even from absorbing all this information even if provided by the system so either if provided by the system so either if provided by the system so either we're getting model as an explanation we're getting model as an explanation we're getting model as an explanation for what's happening and that's it's not for what's happening and that's it's not for what's happening and that's it's not comprehensible to us or we getting a comprehensible to us or we getting a comprehensible to us or we getting a compressed explanation lossy compression compressed explanation lossy compression compressed explanation lossy compression where here's top 10 reasons you got where here's top 10 reasons you got where here's top 10 reasons you got fired it's something but it's not a full fired it's something but it's not a full fired it's something but it's not a full picture you've given elsewhere an picture you've given elsewhere an picture you've given elsewhere an example of of a child and everybody all example of of a child and everybody all example of of a child and everybody all all humans try to deceive they try to all humans try to deceive they try to all humans try to deceive they try to lie early on in their life I think we'll lie early on in their life I think we'll lie early on in their life I think we'll just get a lot of examples of deceptions just get a lot of examples of deceptions just get a lot of examples of deceptions from large language models or AI systems from large language models or AI systems from large language models or AI systems they're going to be kind of shitty or they're going to be kind of shitty or they're going to be kind of shitty or they'll be pretty good but we'll catch they'll be pretty good but we'll catch they'll be pretty good but we'll catch them off guard will start to see the them off guard will start to see the them off guard will start to see the kind of momentum towards uh kind of momentum towards uh kind of momentum towards uh developing increasing deception developing increasing deception developing increasing deception capabilities and that's when you're like capabilities and that's when you're like capabilities and that's when you're like okay we need to do some kind of

okay we need to do some kind of okay we need to do some kind of alignment that prevents deception but alignment that prevents deception but alignment that prevents deception but then we'll have if you support open then we'll have if you support open then we'll have if you support open source then you can have open source source then you can have open source source then you can have open source models that have some level of deception models that have some level of deception models that have some level of deception you can start to explore on a large you can start to explore on a large you can start to explore on a large scale how do we stop it from being scale how do we stop it from being scale how do we stop it from being deceptive then there's a more explicit deceptive then there's a more explicit deceptive then there's a more explicit pragmatic kind of uh problem to solve pragmatic kind of uh problem to solve pragmatic kind of uh problem to solve how do we stop AI systems from uh trying how do we stop AI systems from uh trying how do we stop AI systems from uh trying to optimize for deception that's just an to optimize for deception that's just an to optimize for deception that's just an example right so there is a paper I example right so there is a paper I example right so there is a paper I think it came out last week by Dr parkol think it came out last week by Dr parkol think it came out last week by Dr parkol from MIT I think and they showed that from MIT I think and they showed that from MIT I think and they showed that existing models already showed existing models already showed existing models already showed successful deception in what they do uh successful deception in what they do uh successful deception in what they do uh my concern is not that they lie now and my concern is not that they lie now and my concern is not that they lie now and we need to catch them and tell them we need to catch them and tell them we need to catch them and tell them don't lie my concern is that once they don't lie my concern is that once they don't lie my concern is that once they are capable and deployed they will later are capable and deployed they will later are capable and deployed they will later change their mind because that's what change their mind because that's what change their mind because that's what unrestricted learning allows you to do unrestricted learning allows you to do unrestricted learning allows you to do lots of people grow up maybe in the lots of people grow up maybe in the lots of people grow up maybe in the religious family they read some new religious family they read some new religious family they read some new books and they turn in their books and they turn in their books and they turn in their religion that's a treacherous turn in religion that's a treacherous turn in religion that's a treacherous turn in humans if you learn something new about humans if you learn something new about humans if you learn something new about your colleagues maybe you'll change how your colleagues maybe you'll change how your colleagues maybe you'll change how you react to them yeah the treasures you react to them yeah the treasures you react to them yeah the treasures turn turn turn um if we just mention humans Stalin and um if we just mention humans Stalin and um if we just mention humans Stalin and Hitler there's a turn Stalin is a good Hitler there's a turn Stalin is a good Hitler there's a turn Stalin is a good example he just seems like an normal example he just seems like an normal example he just seems like an normal communist follower Lenin until there's a communist follower Lenin until there's a communist follower Lenin until there's a turn there's a turn of what that means

turn there's a turn of what that means turn there's a turn of what that means in terms of uh when he has complete in terms of uh when he has complete in terms of uh when he has complete control what that what the execution of control what that what the execution of control what that what the execution of that policy means and how many people that policy means and how many people that policy means and how many people get to suffer and you can't say they are get to suffer and you can't say they are get to suffer and you can't say they are not rational the rational decision not rational the rational decision not rational the rational decision changes based on your position then you changes based on your position then you changes based on your position then you are under the boss the r policy maybe to are under the boss the r policy maybe to are under the boss the r policy maybe to be following orders and being honest be following orders and being honest be following orders and being honest when you become a boss rational policy when you become a boss rational policy when you become a boss rational policy May shift yeah and and by the way a lot May shift yeah and and by the way a lot May shift yeah and and by the way a lot of my disagreements here is just to uh of my disagreements here is just to uh of my disagreements here is just to uh Playing devil's advocate to challenge Playing devil's advocate to challenge Playing devil's advocate to challenge your ideas and to explore them together your ideas and to explore them together your ideas and to explore them together so um one of the big problems here in so um one of the big problems here in so um one of the big problems here in this whole conversation this whole conversation this whole conversation is human civilization hangs in the is human civilization hangs in the is human civilization hangs in the balance and yet it's everything is balance and yet it's everything is balance and yet it's everything is unpredictable we don't know how these unpredictable we don't know how these unpredictable we don't know how these systems will look like the robots are coming there's a the robots are coming there's a refrigerator making a buzzing refrigerator making a buzzing refrigerator making a buzzing noise menacing very noise menacing very noise menacing very menacing so every time I'm about to talk menacing so every time I'm about to talk menacing so every time I'm about to talk about this topic things start to happen about this topic things start to happen about this topic things start to happen my flight yesterday was cancelled my flight yesterday was cancelled my flight yesterday was cancelled without possibility to rebook yeah I was without possibility to rebook yeah I was without possibility to rebook yeah I was giving a talk uh at Google in uh Israel giving a talk uh at Google in uh Israel giving a talk uh at Google in uh Israel and uh three cars which were supposed to and uh three cars which were supposed to and uh three cars which were supposed to take me to the talk could not I'm just take me to the talk could not I'm just take me to the talk could not I'm just saying I mean saying I mean saying I mean it I like a eyes I for one welcome our it I like a eyes I for one welcome our it I like a eyes I for one welcome our overlords there's a degree to which we I overlords there's a degree to which we I overlords there's a degree to which we I mean it is very mean it is very mean it is very obvious as we already have we've obvious as we already have we've obvious as we already have we've increasingly given our life over to increasingly given our life over to increasingly given our life over to software

software software systems and then it seems obvious given systems and then it seems obvious given systems and then it seems obvious given the capabilities of AI that are coming the capabilities of AI that are coming the capabilities of AI that are coming that we'll give our lives over that we'll give our lives over that we'll give our lives over increasingly to AI systems cars will increasingly to AI systems cars will increasingly to AI systems cars will drive themselves ref refrigerator drive themselves ref refrigerator drive themselves ref refrigerator eventually will eventually will eventually will optimize uh what I get to optimize uh what I get to optimize uh what I get to eat eat eat and as more and more of our lives are and as more and more of our lives are and as more and more of our lives are controlled or managed by AI assistants controlled or managed by AI assistants controlled or managed by AI assistants it is very possible that there's a drift it is very possible that there's a drift it is very possible that there's a drift I mean I mean I personally am concerned I mean I mean I personally am concerned I mean I mean I personally am concerned about non-existential about non-existential about non-existential stuff the more near-term things because stuff the more near-term things because stuff the more near-term things because before we even get to existential I feel before we even get to existential I feel before we even get to existential I feel like there could be just so many Brave like there could be just so many Brave like there could be just so many Brave New World type of situations you New World type of situations you New World type of situations you mentioned sort of the the term mentioned sort of the the term mentioned sort of the the term behavioral drift the slow behavioral drift the slow behavioral drift the slow boiling that I'm really concerned about boiling that I'm really concerned about boiling that I'm really concerned about as we give our lives over to as we give our lives over to as we give our lives over to automation that our minds can become automation that our minds can become automation that our minds can become controlled by governments by controlled by governments by controlled by governments by companies or just in a distributed way companies or just in a distributed way companies or just in a distributed way there's a there's a there's a drift some aspect of our human nature drift some aspect of our human nature drift some aspect of our human nature gives ourselves over to the control of gives ourselves over to the control of gives ourselves over to the control of AI systems and they in an unintended way AI systems and they in an unintended way AI systems and they in an unintended way just control how we think maybe there'll just control how we think maybe there'll just control how we think maybe there'll be a herd like mentality and how we be a herd like mentality and how we be a herd like mentality and how we think which will kill all creativity and think which will kill all creativity and think which will kill all creativity and exploration of ideas the diversity of exploration of ideas the diversity of exploration of ideas the diversity of ideas or there or or or or much worse so ideas or there or or or or much worse so ideas or there or or or or much worse so it's true it's true but I a lot of the it's true it's true but I a lot of the it's true it's true but I a lot of the uh conversation I'm having you with you uh conversation I'm having you with you uh conversation I'm having you with you now is also kind of wondering almost on now is also kind of wondering almost on now is also kind of wondering almost on a technical level how can AI Escape

a technical level how can AI Escape a technical level how can AI Escape control like what would that system look control like what would that system look control like what would that system look like because it to me is terrifying and like because it to me is terrifying and like because it to me is terrifying and fascinating and also fascinating to me fascinating and also fascinating to me fascinating and also fascinating to me is uh is uh is uh maybe the optimistic notion that it's maybe the optimistic notion that it's maybe the optimistic notion that it's possible to engineer systems that defend possible to engineer systems that defend possible to engineer systems that defend against against against that um one of the things you write a that um one of the things you write a that um one of the things you write a lot about in your book is lot about in your book is lot about in your book is verifiers so not humans humans are also verifiers so not humans humans are also verifiers so not humans humans are also verifiers but software systems that look verifiers but software systems that look verifiers but software systems that look at AI systems and like help you at AI systems and like help you at AI systems and like help you understand this thing is getting real understand this thing is getting real understand this thing is getting real weird help you help you analyze those weird help you help you analyze those weird help you help you analyze those systems so maybe that's a this is a good systems so maybe that's a this is a good systems so maybe that's a this is a good time to talk about verification what is time to talk about verification what is time to talk about verification what is this beautiful notion of verification my this beautiful notion of verification my this beautiful notion of verification my claim is again that there are very claim is again that there are very claim is again that there are very strong limits in what we can and cannot strong limits in what we can and cannot strong limits in what we can and cannot verify uh a lot of times when you post verify uh a lot of times when you post verify uh a lot of times when you post something on social media people go oh I something on social media people go oh I something on social media people go oh I need citation to a peer-reviewed article need citation to a peer-reviewed article need citation to a peer-reviewed article but what is a peer-reviewed article you but what is a peer-reviewed article you but what is a peer-reviewed article you found two people in a world of hundreds found two people in a world of hundreds found two people in a world of hundreds of thousands of scientists who said I of thousands of scientists who said I of thousands of scientists who said I would ever publish it I don't care would ever publish it I don't care would ever publish it I don't care that's the verifier of that process when that's the verifier of that process when that's the verifier of that process when people say oh it's formally verified people say oh it's formally verified people say oh it's formally verified software mathematical proof they accept software mathematical proof they accept software mathematical proof they accept something close to something close to something close to 100% chance of it being free of all 100% chance of it being free of all 100% chance of it being free of all problems but if you actually look at uh problems but if you actually look at uh problems but if you actually look at uh research software is full of bugs old research software is full of bugs old research software is full of bugs old mathematical theorems which been proven mathematical theorems which been proven mathematical theorems which been proven for hundreds of years have been for hundreds of years have been for hundreds of years have been discovered to contain bugs on top of

discovered to contain bugs on top of discovered to contain bugs on top of which we generate new proofs and now we which we generate new proofs and now we which we generate new proofs and now we have to redo all that so verifiers are have to redo all that so verifiers are have to redo all that so verifiers are not perfect usually they are either a not perfect usually they are either a not perfect usually they are either a single human or community pie of humans single human or community pie of humans single human or community pie of humans and it's basically kind of like a and it's basically kind of like a and it's basically kind of like a democratic vote community of democratic vote community of democratic vote community of mathematicians agrees that this proof is mathematicians agrees that this proof is mathematicians agrees that this proof is correct mostly correct even today we're correct mostly correct even today we're correct mostly correct even today we're starting to see some mathematical proofs starting to see some mathematical proofs starting to see some mathematical proofs as so complex so large that mathematical as so complex so large that mathematical as so complex so large that mathematical Community is unable to make a decision Community is unable to make a decision Community is unable to make a decision It looks interesting looks promising but It looks interesting looks promising but It looks interesting looks promising but they don't know they will need years for they don't know they will need years for they don't know they will need years for top Scholars to study it to figure it top Scholars to study it to figure it top Scholars to study it to figure it out so of course we can use AI to help out so of course we can use AI to help out so of course we can use AI to help us with this process but AI is a piece us with this process but AI is a piece us with this process but AI is a piece of software which needs to be verified of software which needs to be verified of software which needs to be verified just to to clarify so verification is just to to clarify so verification is just to to clarify so verification is the process of saying something is the process of saying something is the process of saying something is correct s of the most formal a correct s of the most formal a correct s of the most formal a mathematical proof where there's a mathematical proof where there's a mathematical proof where there's a statement and a series of logical statement and a series of logical statement and a series of logical statements that prove that statement to statements that prove that statement to statements that prove that statement to be correct this is a theorem and you're be correct this is a theorem and you're be correct this is a theorem and you're saying it gets so complex that it's saying it gets so complex that it's saying it gets so complex that it's possible for the human verifiers the possible for the human verifiers the possible for the human verifiers the human beings that verify that the human beings that verify that the human beings that verify that the logical step there's no bugs in it it be logical step there's no bugs in it it be logical step there's no bugs in it it be it becomes a possible so it's nice to it becomes a possible so it's nice to it becomes a possible so it's nice to talk about verification in this most talk about verification in this most talk about verification in this most formal most clear most formal most clear most formal most clear most rigorous formulation of it which is rigorous formulation of it which is rigorous formulation of it which is mathematical proofs right and for AI we mathematical proofs right and for AI we mathematical proofs right and for AI we would like to have that level of would like to have that level of would like to have that level of confidence for very important Mission confidence for very important Mission confidence for very important Mission critical software controlling satellites critical software controlling satellites critical software controlling satellites nuclear power plants for small

nuclear power plants for small nuclear power plants for small deterministic programs we can do this we deterministic programs we can do this we deterministic programs we can do this we can check that code verifies its mapping can check that code verifies its mapping can check that code verifies its mapping to the design whatever software to the design whatever software to the design whatever software Engineers intend it was correctly Engineers intend it was correctly Engineers intend it was correctly implemented but we don't know how to do implemented but we don't know how to do implemented but we don't know how to do this for software which keeps learning this for software which keeps learning this for software which keeps learning self-modifying rewriting its own code we self-modifying rewriting its own code we self-modifying rewriting its own code we don't know how to prove things about the don't know how to prove things about the don't know how to prove things about the physical world states of humans in the physical world states of humans in the physical world states of humans in the physical world so there are papers physical world so there are papers physical world so there are papers coming out now and I have this beautiful coming out now and I have this beautiful coming out now and I have this beautiful one uh towards uh guaranteed safe AI mhm one uh towards uh guaranteed safe AI mhm one uh towards uh guaranteed safe AI mhm very cool paper some of the best authors very cool paper some of the best authors very cool paper some of the best authors uh I ever seen I think there is multiple uh I ever seen I think there is multiple uh I ever seen I think there is multiple touring Award winners there is uh quite touring Award winners there is uh quite touring Award winners there is uh quite you can have this one and one just came you can have this one and one just came you can have this one and one just came out kind of similar uh managing extreme out kind of similar uh managing extreme out kind of similar uh managing extreme AI risks so all of them uh expect this AI risks so all of them uh expect this AI risks so all of them uh expect this level of proof but um I I would say that level of proof but um I I would say that level of proof but um I I would say that uh we can get more confidence with more uh we can get more confidence with more uh we can get more confidence with more resources we put into it but at the end resources we put into it but at the end resources we put into it but at the end of the day we're still as reliable as of the day we're still as reliable as of the day we're still as reliable as the verifiers and you have this infinite the verifiers and you have this infinite the verifiers and you have this infinite regress of verifiers the software used regress of verifiers the software used regress of verifiers the software used to verify a program is itself a piece of to verify a program is itself a piece of to verify a program is itself a piece of program if aliens give us well aligned program if aliens give us well aligned program if aliens give us well aligned super intelligence we can use that to super intelligence we can use that to super intelligence we can use that to create our own safe AI but it's a cat22 create our own safe AI but it's a cat22 create our own safe AI but it's a cat22 you need to have already proven to be you need to have already proven to be you need to have already proven to be safe system to verify this new system of safe system to verify this new system of safe system to verify this new system of equal or greater complexity you just equal or greater complexity you just equal or greater complexity you just mentioned this paper towards guarantee mentioned this paper towards guarantee mentioned this paper towards guarantee safe AI a framework for ensuring robust

safe AI a framework for ensuring robust safe AI a framework for ensuring robust and reliable AI systems like you and reliable AI systems like you and reliable AI systems like you mentioned it's like a who's who Josh mentioned it's like a who's who Josh mentioned it's like a who's who Josh tound yosha Benjo s Russell Max techmar tound yosha Benjo s Russell Max techmar tound yosha Benjo s Russell Max techmar many many many other billion people the many many many other billion people the many many many other billion people the page you have it open on there are many page you have it open on there are many page you have it open on there are many possible strategies for creating safety possible strategies for creating safety possible strategies for creating safety specifications these strategies can specifications these strategies can specifications these strategies can roughly be placed on a spectrum roughly be placed on a spectrum roughly be placed on a spectrum depending on how much safety it would depending on how much safety it would depending on how much safety it would Grant if successfully implemented one Grant if successfully implemented one Grant if successfully implemented one way to do this is as follows and there's way to do this is as follows and there's way to do this is as follows and there's a set of levels from Level zero no a set of levels from Level zero no a set of levels from Level zero no safety specification is used to level safety specification is used to level safety specification is used to level seven the safety specification seven the safety specification seven the safety specification completely encodes all things that completely encodes all things that completely encodes all things that humans might want in all context where humans might want in all context where humans might want in all context where does this paper fall short to you so does this paper fall short to you so does this paper fall short to you so when I wrote a paper artificial when I wrote a paper artificial when I wrote a paper artificial intelligence safety engineering which intelligence safety engineering which intelligence safety engineering which kind of coins the term AI safety that kind of coins the term AI safety that kind of coins the term AI safety that was 2011 we had 2012 conference 2013 was 2011 we had 2012 conference 2013 was 2011 we had 2012 conference 2013 Journal paper one of the things I Journal paper one of the things I Journal paper one of the things I proposed let's just do formal proposed let's just do formal proposed let's just do formal verifications on it let's do verifications on it let's do verifications on it let's do mathematical formal proofs in the mathematical formal proofs in the mathematical formal proofs in the follow-up work I basically realized it follow-up work I basically realized it follow-up work I basically realized it will still not get us 100% we can get will still not get us 100% we can get will still not get us 100% we can get 99.9 we can put more resources 99.9 we can put more resources 99.9 we can put more resources exponentially and get closer but we exponentially and get closer but we exponentially and get closer but we never get to 100% if a system makes a never get to 100% if a system makes a never get to 100% if a system makes a billion decisions a second and you use billion decisions a second and you use billion decisions a second and you use it for 100 years you're still going to it for 100 years you're still going to it for 100 years you're still going to deal with a problem this is wonderful deal with a problem this is wonderful deal with a problem this is wonderful research I'm so happy they doing it this research I'm so happy they doing it this research I'm so happy they doing it this is great but it is not going to be a is great but it is not going to be a is great but it is not going to be a permanent solution to to that problem so permanent solution to to that problem so permanent solution to to that problem so just to clarify the task of creating an just to clarify the task of creating an just to clarify the task of creating an AI verifier is what is creating a

AI verifier is what is creating a AI verifier is what is creating a verifier that the AI system does exactly verifier that the AI system does exactly verifier that the AI system does exactly as it says it does or or it sticks as it says it does or or it sticks as it says it does or or it sticks within the guard rails that it says as within the guard rails that it says as within the guard rails that it says as it must there are many many levels so it must there are many many levels so it must there are many many levels so first you're verifying the hardware in first you're verifying the hardware in first you're verifying the hardware in which it is run you need to verify you which it is run you need to verify you which it is run you need to verify you know Communication channel with the know Communication channel with the know Communication channel with the human you every aspect of that whole human you every aspect of that whole human you every aspect of that whole world model needs to be verified somehow world model needs to be verified somehow world model needs to be verified somehow it needs to map the world into the world it needs to map the world into the world it needs to map the world into the world mble uh map and territory differences so mble uh map and territory differences so mble uh map and territory differences so how do I know internal states of humans how do I know internal states of humans how do I know internal states of humans are you happy or sad I can't tell so how are you happy or sad I can't tell so how are you happy or sad I can't tell so how do I make proofs about real physical do I make proofs about real physical do I make proofs about real physical world yeah I can verify that world yeah I can verify that world yeah I can verify that deterministic algorithm follows certain deterministic algorithm follows certain deterministic algorithm follows certain properties that can be done some people properties that can be done some people properties that can be done some people argue that maybe just maybe 2 plus 2 is argue that maybe just maybe 2 plus 2 is argue that maybe just maybe 2 plus 2 is not four I'm not that not four I'm not that not four I'm not that extreme but once you have sufficiently extreme but once you have sufficiently extreme but once you have sufficiently large proof over sufficiently complex large proof over sufficiently complex large proof over sufficiently complex environment the probability that it has environment the probability that it has environment the probability that it has zero bugs in it is greatly reduced if zero bugs in it is greatly reduced if zero bugs in it is greatly reduced if you keep deploying this a lot eventually you keep deploying this a lot eventually you keep deploying this a lot eventually you going to have a bug anyways there's you going to have a bug anyways there's you going to have a bug anyways there's always a bug there's always a bug and always a bug there's always a bug and always a bug there's always a bug and the fundamental difference is what I the fundamental difference is what I the fundamental difference is what I mentioned we're not dealing with cyber mentioned we're not dealing with cyber mentioned we're not dealing with cyber security we're not going to get a new security we're not going to get a new security we're not going to get a new credit card new Humanity so this paper credit card new Humanity so this paper credit card new Humanity so this paper is really is really is really interesting you said 2011 artificial interesting you said 2011 artificial interesting you said 2011 artificial intelligence safety engineering why intelligence safety engineering why intelligence safety engineering why machine ethics is a wrong machine ethics is a wrong machine ethics is a wrong approach uh the Grand Challenge you approach uh the Grand Challenge you approach uh the Grand Challenge you write of AI safety engineering we write of AI safety engineering we write of AI safety engineering we propose the problem of developing safety propose the problem of developing safety propose the problem of developing safety mechanisms for self-improving

mechanisms for self-improving mechanisms for self-improving systems self-improving systems by the systems self-improving systems by the systems self-improving systems by the way that's an interesting term for the way that's an interesting term for the way that's an interesting term for the thing that we're talking about about is self-improving more General about is self-improving more General than than than learning so self-improving that's an learning so self-improving that's an learning so self-improving that's an interesting term you can improve the interesting term you can improve the interesting term you can improve the rate at which you are learning you can rate at which you are learning you can rate at which you are learning you can become more efficient meta become more efficient meta become more efficient meta Optimizer the word self it's like Optimizer the word self it's like Optimizer the word self it's like self-replicating self-replicating self-replicating self-improving you can imagine a system self-improving you can imagine a system self-improving you can imagine a system building its own world on a building its own world on a building its own world on a scale and in a way that is way different scale and in a way that is way different scale and in a way that is way different than the current systems do it feels than the current systems do it feels than the current systems do it feels like the current systems are not like the current systems are not like the current systems are not self-improving or self-replicating or self-improving or self-replicating or self-improving or self-replicating or self- growing or self spreading all that self- growing or self spreading all that self- growing or self spreading all that kind of stuff and once you take that kind of stuff and once you take that kind of stuff and once you take that leap that's when a lot of the challenges leap that's when a lot of the challenges leap that's when a lot of the challenges seems to happen because the kind of bugs seems to happen because the kind of bugs seems to happen because the kind of bugs you can find you can find you can find now seems more akin to the current sort now seems more akin to the current sort now seems more akin to the current sort of normal of normal of normal software debugging kind of software debugging kind of software debugging kind of process uh but whenever you can do process uh but whenever you can do process uh but whenever you can do self-replication and arbitrary self-replication and arbitrary self-replication and arbitrary self-improvement that's when a bug can self-improvement that's when a bug can self-improvement that's when a bug can become a real problem real real become a real problem real real become a real problem real real fast uh so what is the difference to you fast uh so what is the difference to you fast uh so what is the difference to you between verification of a non between verification of a non between verification of a non self-improving system versus a self-improving system versus a self-improving system versus a verification of a self-improving system verification of a self-improving system verification of a self-improving system so if you have fixed code for example so if you have fixed code for example so if you have fixed code for example you can verify that code static you can verify that code static you can verify that code static verification at the time but if it will

verification at the time but if it will verification at the time but if it will continue modifying it you have a much continue modifying it you have a much continue modifying it you have a much harder time guaranteeing that important harder time guaranteeing that important harder time guaranteeing that important properties of that system have not been properties of that system have not been properties of that system have not been modified then the code changed is it modified then the code changed is it modified then the code changed is it even doable no does the does the whole even doable no does the does the whole even doable no does the does the whole process of verification just completely process of verification just completely process of verification just completely fall apart it can always cheat it can fall apart it can always cheat it can fall apart it can always cheat it can store parts of its code outside in the store parts of its code outside in the store parts of its code outside in the environment it can have kind of extended environment it can have kind of extended environment it can have kind of extended mind situation so this is exactly the mind situation so this is exactly the mind situation so this is exactly the type of problems I'm trying to bring up type of problems I'm trying to bring up type of problems I'm trying to bring up what are the classes of verifiers that what are the classes of verifiers that what are the classes of verifiers that you write about in the book is there you write about in the book is there you write about in the book is there interesting ones that stand out to you interesting ones that stand out to you interesting ones that stand out to you you have your some favorites so I like you have your some favorites so I like you have your some favorites so I like Oracle types where you kind of just know Oracle types where you kind of just know Oracle types where you kind of just know that it's right touring lik Oracle that it's right touring lik Oracle that it's right touring lik Oracle machines they know the right answer how machines they know the right answer how machines they know the right answer how who knows but they pull it out from who knows but they pull it out from who knows but they pull it out from somewhere so you have to trust them and somewhere so you have to trust them and somewhere so you have to trust them and that's a concern I have about humans uh that's a concern I have about humans uh that's a concern I have about humans uh in a world with very smart machines we in a world with very smart machines we in a world with very smart machines we experiment with them we see after a experiment with them we see after a experiment with them we see after a while okay they always been right before while okay they always been right before while okay they always been right before and we start trusting them without any and we start trusting them without any and we start trusting them without any verification of what they are saying oh verification of what they are saying oh verification of what they are saying oh I see that we kind of build Oracle I see that we kind of build Oracle I see that we kind of build Oracle verifiers or rather we build verifiers verifiers or rather we build verifiers verifiers or rather we build verifiers we believe to be we believe to be we believe to be oracles and then we start to without any oracles and then we start to without any oracles and then we start to without any proof use them as if they're Oracle proof use them as if they're Oracle proof use them as if they're Oracle verifi we remove ourselves from that verifi we remove ourselves from that verifi we remove ourselves from that process we are not scientists who process we are not scientists who process we are not scientists who understand the world we are humans who understand the world we are humans who understand the world we are humans who get new data presented to us okay one get new data presented to us okay one get new data presented to us okay one one really cool class of air fires is a one really cool class of air fires is a one really cool class of air fires is a self aif fire is it possible that you

self aif fire is it possible that you self aif fire is it possible that you somehow engineer into AI systems that somehow engineer into AI systems that somehow engineer into AI systems that think that constantly verifies itself think that constantly verifies itself think that constantly verifies itself preserved portion of it can be done but preserved portion of it can be done but preserved portion of it can be done but in terms of in terms of in terms of mathematical verification it's kind of mathematical verification it's kind of mathematical verification it's kind of useless you saying you are the greatest useless you saying you are the greatest useless you saying you are the greatest guy in the world because you are saying guy in the world because you are saying guy in the world because you are saying it it's circular and not very helpful it it's circular and not very helpful it it's circular and not very helpful but it's consistent we know that within but it's consistent we know that within but it's consistent we know that within that world you have verified that system that world you have verified that system that world you have verified that system in a paper I try to kind of brute force in a paper I try to kind of brute force in a paper I try to kind of brute force all possible verifiers it doesn't mean all possible verifiers it doesn't mean all possible verifiers it doesn't mean that this one particularly important to that this one particularly important to that this one particularly important to us but what about like us but what about like us but what about like self-doubt like the kind of verification self-doubt like the kind of verification self-doubt like the kind of verification where you said you say or I say I'm the where you said you say or I say I'm the where you said you say or I say I'm the greatest guy in the world what about a greatest guy in the world what about a greatest guy in the world what about a thing which I actually have is is a thing which I actually have is is a thing which I actually have is is a voice that is constantly extremely voice that is constantly extremely voice that is constantly extremely critical so critical so critical so like engineer into the system a constant like engineer into the system a constant like engineer into the system a constant uncertainty about uncertainty about uncertainty about self a constant doubt well any smart self a constant doubt well any smart self a constant doubt well any smart system would have doubt about everything system would have doubt about everything system would have doubt about everything right you not sure if what information right you not sure if what information right you not sure if what information you are given is through if you are you are given is through if you are you are given is through if you are subject to manipulation subject to manipulation subject to manipulation you have this Safety and Security you have this Safety and Security you have this Safety and Security mindset but I mean you have doubt about mindset but I mean you have doubt about mindset but I mean you have doubt about yourself so the AI yourself so the AI yourself so the AI systems that has a doubt about whether systems that has a doubt about whether systems that has a doubt about whether the thing is doing is causing harm is the thing is doing is causing harm is the thing is doing is causing harm is the right thing to be doing so just a the right thing to be doing so just a the right thing to be doing so just a constant doubt about what it's doing constant doubt about what it's doing constant doubt about what it's doing because it's hard to be a dictator full because it's hard to be a dictator full because it's hard to be a dictator full of doubt I I may be wrong but I think of doubt I I may be wrong but I think of doubt I I may be wrong but I think steuart Russell's uh ideas are all about

steuart Russell's uh ideas are all about steuart Russell's uh ideas are all about machines which are un certain about what machines which are un certain about what machines which are un certain about what humans want and trying to learn better humans want and trying to learn better humans want and trying to learn better and better what we want the problem of and better what we want the problem of and better what we want the problem of course is we don't know what we want and course is we don't know what we want and course is we don't know what we want and we don't agree on it yeah but we don't agree on it yeah but we don't agree on it yeah but uncertainty his his idea is that having uncertainty his his idea is that having uncertainty his his idea is that having that like uh self-doubt uncertainty in that like uh self-doubt uncertainty in that like uh self-doubt uncertainty in AI systems engineered into AI systems is AI systems engineered into AI systems is AI systems engineered into AI systems is one way to solve the control problem it one way to solve the control problem it one way to solve the control problem it could also backfire maybe you uncertain could also backfire maybe you uncertain could also backfire maybe you uncertain about completing your mission like I am about completing your mission like I am about completing your mission like I am paranoid about your camera is not paranoid about your camera is not paranoid about your camera is not recording right now so I would feel much recording right now so I would feel much recording right now so I would feel much better if you had a secondary camera but better if you had a secondary camera but better if you had a secondary camera but I also would feel even better if you had I also would feel even better if you had I also would feel even better if you had a third and eventually I would turn this a third and eventually I would turn this a third and eventually I would turn this whole world into cameras pointing at us whole world into cameras pointing at us whole world into cameras pointing at us making sure we're capturing this no but making sure we're capturing this no but making sure we're capturing this no but wouldn't you have a meta wouldn't you have a meta wouldn't you have a meta concern like that you just stated that concern like that you just stated that concern like that you just stated that eventually there'll be way too many eventually there'll be way too many eventually there'll be way too many cameras so you would be able to keep cameras so you would be able to keep cameras so you would be able to keep zooming on in the big zooming on in the big zooming on in the big picture of your picture of your picture of your concerns so it's a multi-objective concerns so it's a multi-objective concerns so it's a multi-objective optimization it depends how much I value optimization it depends how much I value optimization it depends how much I value capturing this versus not destroying the capturing this versus not destroying the capturing this versus not destroying the universe right universe right universe right exactly and and then you will also ask exactly and and then you will also ask exactly and and then you will also ask about like what does it mean to destroy about like what does it mean to destroy about like what does it mean to destroy the universe and how many universes are the universe and how many universes are the universe and how many universes are and you keep asking that question but and you keep asking that question but and you keep asking that question but that doubting yourself would prevent you that doubting yourself would prevent you that doubting yourself would prevent you from destroying the universe because from destroying the universe because from destroying the universe because you're constantly full of doubt it might you're constantly full of doubt it might you're constantly full of doubt it might affect your productivity just you might affect your productivity just you might affect your productivity just you might be scared to do anything it's scared to be scared to do anything it's scared to be scared to do anything it's scared to do anything mess things up well that's do anything mess things up well that's do anything mess things up well that's better I mean I guess the question is it

better I mean I guess the question is it better I mean I guess the question is it possible to engineer that in I guess possible to engineer that in I guess possible to engineer that in I guess your answer would be yes but we don't your answer would be yes but we don't your answer would be yes but we don't know how to do that and we need to know how to do that and we need to know how to do that and we need to invest a lot of effort into figuring out invest a lot of effort into figuring out invest a lot of effort into figuring out how to do that but it's unlikely how to do that but it's unlikely how to do that but it's unlikely underpinning a lot of your writing is underpinning a lot of your writing is underpinning a lot of your writing is this sense that we're this sense that we're this sense that we're screwed but it just feels like it's an screwed but it just feels like it's an screwed but it just feels like it's an engineering problem I don't understand engineering problem I don't understand engineering problem I don't understand why we're screwed it it we time and time why we're screwed it it we time and time why we're screwed it it we time and time again Humanity has gotten itself into again Humanity has gotten itself into again Humanity has gotten itself into trouble and figured out a way to get out trouble and figured out a way to get out trouble and figured out a way to get out of the trouble we are in a situation of the trouble we are in a situation of the trouble we are in a situation where people making more capable systems where people making more capable systems where people making more capable systems just need more just need more just need more resources they don't need to invent resources they don't need to invent resources they don't need to invent anything in my opinion some will anything in my opinion some will anything in my opinion some will disagree but so far at least I don't see disagree but so far at least I don't see disagree but so far at least I don't see diminishing returns if you have 10x diminishing returns if you have 10x diminishing returns if you have 10x compute you'll get better performance compute you'll get better performance compute you'll get better performance the same doesn't apply to safety if you the same doesn't apply to safety if you the same doesn't apply to safety if you give uh Mei or any other organization 10 give uh Mei or any other organization 10 give uh Mei or any other organization 10 times the money they don't output 10 times the money they don't output 10 times the money they don't output 10 times the safety and the Gap be between times the safety and the Gap be between times the safety and the Gap be between capabilities and safety becomes bigger capabilities and safety becomes bigger capabilities and safety becomes bigger and bigger all the time so it's hard to and bigger all the time so it's hard to and bigger all the time so it's hard to be completely optimistic about our be completely optimistic about our be completely optimistic about our results here I can name 10 excellent results here I can name 10 excellent results here I can name 10 excellent breakthrough papers in machine learning breakthrough papers in machine learning breakthrough papers in machine learning I would struggle to name equally I would struggle to name equally I would struggle to name equally important breakthroughs in safety a lot important breakthroughs in safety a lot important breakthroughs in safety a lot of times a safety paper will propose a of times a safety paper will propose a of times a safety paper will propose a toy solution and point out 10 new toy solution and point out 10 new toy solution and point out 10 new problems discovered as a result it's problems discovered as a result it's problems discovered as a result it's like this fractal you're zooming in and like this fractal you're zooming in and like this fractal you're zooming in and you see more problems and it's infinite you see more problems and it's infinite you see more problems and it's infinite in all directions does this apply to

in all directions does this apply to in all directions does this apply to other Technologies or is this is this other Technologies or is this is this other Technologies or is this is this unique to AI where safety is always unique to AI where safety is always unique to AI where safety is always lagging behind so I guess we can look at lagging behind so I guess we can look at lagging behind so I guess we can look at related Technologies with cyber security related Technologies with cyber security related Technologies with cyber security right we we did manage to have Banks and right we we did manage to have Banks and right we we did manage to have Banks and casinos and Bitcoin so you can have casinos and Bitcoin so you can have casinos and Bitcoin so you can have secure narrow systems which are doing secure narrow systems which are doing secure narrow systems which are doing okay uh narrow attacks on them fail but okay uh narrow attacks on them fail but okay uh narrow attacks on them fail but you can always go outside outside of a you can always go outside outside of a you can always go outside outside of a box so if I I can't hack you Bitcoin I box so if I I can't hack you Bitcoin I box so if I I can't hack you Bitcoin I can hack you so there is always can hack you so there is always can hack you so there is always something if I really want it I will something if I really want it I will something if I really want it I will find a different way we talk about uh find a different way we talk about uh find a different way we talk about uh guard rails for AI well that's a fence I guard rails for AI well that's a fence I guard rails for AI well that's a fence I can dig a tunnel under it I can jump can dig a tunnel under it I can jump can dig a tunnel under it I can jump over it I can climb it I can walk around over it I can climb it I can walk around over it I can climb it I can walk around it you may have a very nice guard rail it you may have a very nice guard rail it you may have a very nice guard rail but in a real world it's not a permanent but in a real world it's not a permanent but in a real world it's not a permanent guarantee of safety and again this is guarantee of safety and again this is guarantee of safety and again this is the fundamental difference we are not the fundamental difference we are not the fundamental difference we are not saying we need to be 90% safe to get saying we need to be 90% safe to get saying we need to be 90% safe to get those trillions of dollars of benefit we those trillions of dollars of benefit we those trillions of dollars of benefit we need to be 100% indefinitely Or we might need to be 100% indefinitely Or we might need to be 100% indefinitely Or we might lose the principle so if you look at lose the principle so if you look at lose the principle so if you look at just uh humanity is a set of just uh humanity is a set of just uh humanity is a set of machines is is the is the Machinery of machines is is the is the Machinery of machines is is the is the Machinery of AI safety uh conflicting with the Machinery safety uh conflicting with the Machinery of capitalism I think we can generalize of capitalism I think we can generalize of capitalism I think we can generalize it to just uh prisoners dilemma in it to just uh prisoners dilemma in it to just uh prisoners dilemma in general personal self-interest versus general personal self-interest versus general personal self-interest versus group group group interest the incentives are such that

interest the incentives are such that interest the incentives are such that everyone wants what's best for them everyone wants what's best for them everyone wants what's best for them capitalism obviously has that tendency capitalism obviously has that tendency capitalism obviously has that tendency to maximize your personal gain uh which to maximize your personal gain uh which to maximize your personal gain uh which does create this race to the bottom I does create this race to the bottom I does create this race to the bottom I don't have to be a lot better than you don't have to be a lot better than you don't have to be a lot better than you but if I'm 1% better than you I'll but if I'm 1% better than you I'll but if I'm 1% better than you I'll capture more of a profit so it's worth capture more of a profit so it's worth capture more of a profit so it's worth for me personally to take the risk even for me personally to take the risk even for me personally to take the risk even if society as a whole will suffer as a if society as a whole will suffer as a if society as a whole will suffer as a result result result so capitalism has created a lot of good so capitalism has created a lot of good so capitalism has created a lot of good in this world it's not clear to me that AI world it's not clear to me that AI safety is not aligned with the function safety is not aligned with the function safety is not aligned with the function of capitalism unless AI safety is so of capitalism unless AI safety is so of capitalism unless AI safety is so difficult that it requires the complete difficult that it requires the complete difficult that it requires the complete halt of the halt of the halt of the development which is also a possibility development which is also a possibility development which is also a possibility it just feels like building Safe Systems it just feels like building Safe Systems it just feels like building Safe Systems should be the desirable thing to do for should be the desirable thing to do for should be the desirable thing to do for tech companies tech companies tech companies right look at um governance structures right look at um governance structures right look at um governance structures then you have someone with complete then you have someone with complete then you have someone with complete power they're extremely dangerous so the power they're extremely dangerous so the power they're extremely dangerous so the solution we came up with is break it up solution we came up with is break it up solution we came up with is break it up you have judicial legislative executive you have judicial legislative executive you have judicial legislative executive same here have narrow AI systems work on same here have narrow AI systems work on same here have narrow AI systems work on important problems solve immortality important problems solve immortality important problems solve immortality it's a biological problem we can solve it's a biological problem we can solve it's a biological problem we can solve similar to how progress was made with similar to how progress was made with similar to how progress was made with protein folding using a system which protein folding using a system which protein folding using a system which doesn't also play doesn't also play doesn't also play chess there is no no reason to create chess there is no no reason to create chess there is no no reason to create super intelligent system to get most of

super intelligent system to get most of super intelligent system to get most of the benefits we want from much safer the benefits we want from much safer the benefits we want from much safer Naro systems it really is a question to Naro systems it really is a question to Naro systems it really is a question to me me me whether companies are interested in whether companies are interested in whether companies are interested in creating anything but n AI I think the creating anything but n AI I think the creating anything but n AI I think the when term AGI is used by tech companies when term AGI is used by tech companies when term AGI is used by tech companies they mean narrow they mean narrow they mean narrow AI they mean narrow AI with amazing AI they mean narrow AI with amazing AI they mean narrow AI with amazing capabilities capabilities capabilities I I do think that there's a leap between I I do think that there's a leap between I I do think that there's a leap between narrow AI with amazing capabilities with narrow AI with amazing capabilities with narrow AI with amazing capabilities with superhuman capabilities and the kind superhuman capabilities and the kind superhuman capabilities and the kind of self-motivated agent like AGI system of self-motivated agent like AGI system of self-motivated agent like AGI system that we're talking about I don't know if that we're talking about I don't know if that we're talking about I don't know if it's obvious to me that a company would it's obvious to me that a company would it's obvious to me that a company would want to take the leap to creating an AGI want to take the leap to creating an AGI want to take the leap to creating an AGI that it would lose control of because that it would lose control of because that it would lose control of because then he can't capture the value from then he can't capture the value from then he can't capture the value from that system but the bragging rights that system but the bragging rights that system but the bragging rights but being first it is the same humans but being first it is the same humans but being first it is the same humans who are in of systems right so that's a who are in of systems right so that's a who are in of systems right so that's a that that jumps from the the incentives that that jumps from the the incentives that that jumps from the the incentives of capitalism to human nature and so of capitalism to human nature and so of capitalism to human nature and so there the question is whether human there the question is whether human there the question is whether human nature will override the interest of the nature will override the interest of the nature will override the interest of the company so you've mentioned slowing or company so you've mentioned slowing or company so you've mentioned slowing or halting halting halting progress is that one possible solution progress is that one possible solution progress is that one possible solution are you proponent of pausing development are you proponent of pausing development are you proponent of pausing development of AI whether it's for six months or of AI whether it's for six months or of AI whether it's for six months or completely completely completely the condition would be not time but

the condition would be not time but the condition would be not time but capabilities pause until you can do XYZ capabilities pause until you can do XYZ capabilities pause until you can do XYZ and if I'm right and you cannot it's and if I'm right and you cannot it's and if I'm right and you cannot it's impossible then it becomes a permanent impossible then it becomes a permanent impossible then it becomes a permanent ban but if you right and it's possible ban but if you right and it's possible ban but if you right and it's possible so as soon as you have those safety so as soon as you have those safety so as soon as you have those safety capabilities go ahead right so is there capabilities go ahead right so is there capabilities go ahead right so is there any actual explicit any actual explicit any actual explicit capabilities that you can put on paper capabilities that you can put on paper capabilities that you can put on paper that we as a human civilization could that we as a human civilization could that we as a human civilization could put on paper is it possible to make it put on paper is it possible to make it put on paper is it possible to make it explicit like that explicit like that explicit like that like uh versus kind of a vague notion of like uh versus kind of a vague notion of like uh versus kind of a vague notion of just like you said it's very vague we just like you said it's very vague we just like you said it's very vague we want to ask system to do good and want want to ask system to do good and want want to ask system to do good and want them to be safe those are very vague them to be safe those are very vague them to be safe those are very vague Notions is there more formal Notions so Notions is there more formal Notions so Notions is there more formal Notions so then I think about this problem I think then I think about this problem I think then I think about this problem I think about having a toolbox I would need about having a toolbox I would need about having a toolbox I would need capabilities such as explaining capabilities such as explaining capabilities such as explaining everything about that systems design and everything about that systems design and everything about that systems design and workings predicting not just terminal workings predicting not just terminal workings predicting not just terminal goal but all the intermediate steps of a goal but all the intermediate steps of a goal but all the intermediate steps of a system control in terms of either Direct system control in terms of either Direct system control in terms of either Direct Control some sort of a hybrid option Control some sort of a hybrid option Control some sort of a hybrid option ideal advisor doesn't matter which one ideal advisor doesn't matter which one ideal advisor doesn't matter which one you pick but you have to be able to you pick but you have to be able to you pick but you have to be able to achieve it in a book we talk about achieve it in a book we talk about achieve it in a book we talk about others others others verification is another very important verification is another very important verification is another very important tool um communication without ambiguity tool um communication without ambiguity tool um communication without ambiguity human language is ambiguous that's human language is ambiguous that's human language is ambiguous that's another source of danger so another source of danger so another source of danger so basically there is uh a paper we basically there is uh a paper we basically there is uh a paper we published in ACM surveys which looks at published in ACM surveys which looks at published in ACM surveys which looks at about 50 different impossibility results

about 50 different impossibility results about 50 different impossibility results which may or may not be relevant to this which may or may not be relevant to this which may or may not be relevant to this problem but we don't have enough human problem but we don't have enough human problem but we don't have enough human resources to investigate all of them for resources to investigate all of them for resources to investigate all of them for relevance to AI safety the ones I relevance to AI safety the ones I relevance to AI safety the ones I mentioned to you I definitely think mentioned to you I definitely think mentioned to you I definitely think would be handy and that's what we see AI would be handy and that's what we see AI would be handy and that's what we see AI safety researchers working on safety researchers working on safety researchers working on explainability is a huge one the problem explainability is a huge one the problem explainability is a huge one the problem is that it's very hard to separate is that it's very hard to separate is that it's very hard to separate capabilities work from safety work if capabilities work from safety work if capabilities work from safety work if you make good progress in explainability you make good progress in explainability you make good progress in explainability now the system itself can engage in now the system itself can engage in now the system itself can engage in self-improvement much easier increasing self-improvement much easier increasing self-improvement much easier increasing capability greatly so it's not obvious capability greatly so it's not obvious capability greatly so it's not obvious that there is any research which is pure that there is any research which is pure that there is any research which is pure safety work without disproportionate safety work without disproportionate safety work without disproportionate increase in capability and danger increase in capability and danger increase in capability and danger explainability is really interesting um explainability is really interesting um explainability is really interesting um why is that connected to you to why is that connected to you to why is that connected to you to capability if it's able to explain capability if it's able to explain capability if it's able to explain itself well why does that naturally mean itself well why does that naturally mean itself well why does that naturally mean that it's more capable right now it's uh that it's more capable right now it's uh that it's more capable right now it's uh comprised of weights on a neural network comprised of weights on a neural network comprised of weights on a neural network if it can convert it to manipulatable if it can convert it to manipulatable if it can convert it to manipulatable code like software it's a lot easier to code like software it's a lot easier to code like software it's a lot easier to work in work in work in self-improvement I see so it it uh you self-improvement I see so it it uh you self-improvement I see so it it uh you can do intelligent design instead of can do intelligent design instead of can do intelligent design instead of evolutionary gradual descent well you evolutionary gradual descent well you evolutionary gradual descent well you could probably do human feedback human could probably do human feedback human could probably do human feedback human alignment more effectively if it's able alignment more effectively if it's able alignment more effectively if it's able to be explainable if it's able to to be explainable if it's able to to be explainable if it's able to convert the waste into human convert the waste into human convert the waste into human understandable form then you can understandable form then you can understandable form then you can probably have humans interact with it probably have humans interact with it probably have humans interact with it better do you think there's hope that we better do you think there's hope that we better do you think there's hope that we can make AI systems can make AI systems can make AI systems explainable not completely so if they explainable not completely so if they explainable not completely so if they sufficiently

sufficiently sufficiently large you simply don't have the capacity large you simply don't have the capacity large you simply don't have the capacity to comprehend what all the trillions of to comprehend what all the trillions of to comprehend what all the trillions of connections represent again you can connections represent again you can connections represent again you can obviously get a very useful explanation obviously get a very useful explanation obviously get a very useful explanation which talks about top most important which talks about top most important which talks about top most important features which contribute to the features which contribute to the features which contribute to the decision but the only true explanation decision but the only true explanation decision but the only true explanation is the model is the model is the model itself so there's deception be part of itself so there's deception be part of itself so there's deception be part of the explanation right so you can never the explanation right so you can never the explanation right so you can never prove that there's some deception in the prove that there's some deception in the prove that there's some deception in the in the network explaining itself in the network explaining itself in the network explaining itself absolutely and you can probably have absolutely and you can probably have absolutely and you can probably have targeted deception where different targeted deception where different targeted deception where different individuals will understand explanation individuals will understand explanation individuals will understand explanation in different ways based on their in different ways based on their in different ways based on their cognitive capability so while what cognitive capability so while what cognitive capability so while what you're saying may be the same and true you're saying may be the same and true you're saying may be the same and true in some situations others will be in some situations others will be in some situations others will be deceived by it so it's impossible for an deceived by it so it's impossible for an deceived by it so it's impossible for an AI system to be truly fully AI system to be truly fully AI system to be truly fully explainable in the way that we explainable in the way that we explainable in the way that we mean honestly at extreme the systems mean honestly at extreme the systems mean honestly at extreme the systems which are narrow and less complex could which are narrow and less complex could which are narrow and less complex could be understood pretty well if it's be understood pretty well if it's be understood pretty well if it's impossible to be perfectly explainable impossible to be perfectly explainable impossible to be perfectly explainable is there a hopeful perspective on that is there a hopeful perspective on that is there a hopeful perspective on that like it's impossible to be perfectly like it's impossible to be perfectly like it's impossible to be perfectly explainable but you can explain most of explainable but you can explain most of explainable but you can explain most of the important the important the important stuff Mo most you can you can ask a stuff Mo most you can you can ask a stuff Mo most you can you can ask a system what are what are the worst ways system what are what are the worst ways system what are what are the worst ways you can hurt humans and it will answer you can hurt humans and it will answer you can hurt humans and it will answer honestly any work in a safety direction honestly any work in a safety direction honestly any work in a safety direction right now seems like a good idea because right now seems like a good idea because right now seems like a good idea because we are not slowing down I'm we are not slowing down I'm we are not slowing down I'm not for a second thinking that uh my

not for a second thinking that uh my not for a second thinking that uh my message or anyone else's will be heard message or anyone else's will be heard message or anyone else's will be heard and will be a sane civilization which and will be a sane civilization which and will be a sane civilization which decides not to kill itself by creating decides not to kill itself by creating decides not to kill itself by creating its own Replacements the pausing of its own Replacements the pausing of its own Replacements the pausing of development is an impossible thing for development is an impossible thing for development is an impossible thing for you again it's always limited by either you again it's always limited by either you again it's always limited by either Geographic constraints PA in US PA in Geographic constraints PA in US PA in Geographic constraints PA in US PA in China so there are other jurisdictions China so there are other jurisdictions China so there are other jurisdictions as um the scale of a project becomes as um the scale of a project becomes as um the scale of a project becomes smaller so right now it's like Manhattan smaller so right now it's like Manhattan smaller so right now it's like Manhattan Project scale in terms of costs and Project scale in terms of costs and Project scale in terms of costs and people but if 5 years from now Compu is people but if 5 years from now Compu is people but if 5 years from now Compu is available on a desktop to do it available on a desktop to do it available on a desktop to do it regulation will not help you can't regulation will not help you can't regulation will not help you can't control it as easy any kid in a garage control it as easy any kid in a garage control it as easy any kid in a garage can train a model so a lot of it is in can train a model so a lot of it is in can train a model so a lot of it is in my opinion just safety theater security my opinion just safety theater security my opinion just safety theater security theater whereever we saying oh it's theater whereever we saying oh it's theater whereever we saying oh it's illegal to train models so big okay well illegal to train models so big okay well illegal to train models so big okay well so okay that's security theater and is so okay that's security theater and is so okay that's security theater and is government regulation also security government regulation also security government regulation also security theater given that a lot of the terms theater given that a lot of the terms theater given that a lot of the terms are not well defined and uh really are not well defined and uh really are not well defined and uh really cannot be enforced in real life we don't cannot be enforced in real life we don't cannot be enforced in real life we don't have ways to monitor training runs have ways to monitor training runs have ways to monitor training runs meaningfully live while they take place meaningfully live while they take place meaningfully live while they take place there are limits to testing for there are limits to testing for there are limits to testing for capabilities I mentioned so a lot of it capabilities I mentioned so a lot of it capabilities I mentioned so a lot of it cannot be enforced do I strongly support cannot be enforced do I strongly support cannot be enforced do I strongly support all that regulation yes of course any all that regulation yes of course any all that regulation yes of course any type of red tape will slow down and take type of red tape will slow down and take type of red tape will slow down and take money away from compute towards money away from compute towards money away from compute towards lowers can you help me understand what lowers can you help me understand what lowers can you help me understand what is the hopeful path here for you is the hopeful path here for you is the hopeful path here for you solution

solution solution wise out of this it sounds like you're wise out of this it sounds like you're wise out of this it sounds like you're saying AI systems in the end are saying AI systems in the end are saying AI systems in the end are unverifiable unpredictable as the book unverifiable unpredictable as the book unverifiable unpredictable as the book says says says unexplainable um uncontrollable that's unexplainable um uncontrollable that's unexplainable um uncontrollable that's the big one uncontrollable and all the the big one uncontrollable and all the the big one uncontrollable and all the other UNS just make it difficult to other UNS just make it difficult to other UNS just make it difficult to avoid void getting to the uncontrollable avoid void getting to the uncontrollable avoid void getting to the uncontrollable I guess but once it's uncontrollable I guess but once it's uncontrollable I guess but once it's uncontrollable then it just goes goes then it just goes goes then it just goes goes wild surely there's Solutions humans are wild surely there's Solutions humans are wild surely there's Solutions humans are pretty pretty pretty smart what are what are possible smart what are what are possible smart what are what are possible solutions like if you are dictator of solutions like if you are dictator of solutions like if you are dictator of the world what what do we do so the the world what what do we do so the the world what what do we do so the smart thing is not to build something smart thing is not to build something smart thing is not to build something you cannot control you cannot understand you cannot control you cannot understand you cannot control you cannot understand build what you can and benefit from it build what you can and benefit from it build what you can and benefit from it I'm a big believer in personal I'm a big believer in personal I'm a big believer in personal self-interest a lot of the guys running self-interest a lot of the guys running self-interest a lot of the guys running those companies are young rich people those companies are young rich people those companies are young rich people what do they have to gain Beyond what do they have to gain Beyond what do they have to gain Beyond billions they already have financially billions they already have financially billions they already have financially right it's uh not a requirement that right it's uh not a requirement that right it's uh not a requirement that they press that button they can easily they press that button they can easily they press that button they can easily wait a long time they can just choose wait a long time they can just choose wait a long time they can just choose not to do it and still have amazing not to do it and still have amazing not to do it and still have amazing life uh in history a lot of times if you life uh in history a lot of times if you life uh in history a lot of times if you did something really bad at least you did something really bad at least you did something really bad at least you became part of history books there is a became part of history books there is a became part of history books there is a chance in this case there won't be any chance in this case there won't be any chance in this case there won't be any history so you're saying the individuals history so you're saying the individuals history so you're saying the individuals running these companies running these companies running these companies should do some soul searching and and should do some soul searching and and should do some soul searching and and what and stop development well either what and stop development well either what and stop development well either they have to prove that of course it's they have to prove that of course it's they have to prove that of course it's possible to indefinitely control Godlike

possible to indefinitely control Godlike possible to indefinitely control Godlike super intelligent machines by humans and super intelligent machines by humans and super intelligent machines by humans and ideally let us know how or agree that ideally let us know how or agree that ideally let us know how or agree that it's not possible and it's a very bad it's not possible and it's a very bad it's not possible and it's a very bad idea to do it including for them idea to do it including for them idea to do it including for them personally and their families and personally and their families and personally and their families and friends and capital so what do you think friends and capital so what do you think friends and capital so what do you think the actual meetings inside these the actual meetings inside these the actual meetings inside these companies look like don't you think companies look like don't you think companies look like don't you think they're all all the engineers really it they're all all the engineers really it they're all all the engineers really it is the engineers that make this happen is the engineers that make this happen is the engineers that make this happen they're not like automatons they're they're not like automatons they're they're not like automatons they're human beings they're brilliant human human beings they're brilliant human human beings they're brilliant human being so being so being so they're they're non-stop asking how do they're they're non-stop asking how do they're they're non-stop asking how do we make sure this is safe so again I'm we make sure this is safe so again I'm we make sure this is safe so again I'm not inside from outside it seems like not inside from outside it seems like not inside from outside it seems like there is uh certain filtering going on there is uh certain filtering going on there is uh certain filtering going on and restrictions and criticism and what and restrictions and criticism and what and restrictions and criticism and what they can say and everyone who was they can say and everyone who was they can say and everyone who was working in charge of safety and whose working in charge of safety and whose working in charge of safety and whose responsibility it was to protect us said responsibility it was to protect us said responsibility it was to protect us said you know what I'm going home so that's you know what I'm going home so that's you know what I'm going home so that's not encouraging what do you think the not encouraging what do you think the not encouraging what do you think the discussion inside those companies look discussion inside those companies look discussion inside those companies look like you're you're developing you're like you're you're developing you're like you're you're developing you're training GPT 5 you're you're you're training GPT 5 you're you're you're training GPT 5 you're you're you're training Gemini you're training Claude training Gemini you're training Claude training Gemini you're training Claude and and and grock don't you think they're constantly grock don't you think they're constantly grock don't you think they're constantly like underneath it's not maybe it's not like underneath it's not maybe it's not like underneath it's not maybe it's not made explicit but you're constantly sort made explicit but you're constantly sort made explicit but you're constantly sort of wondering like of wondering like of wondering like where uh where do the system currently where uh where do the system currently where uh where do the system currently stand where did the possible un and the stand where did the possible un and the stand where did the possible un and the consequences where are the the the the consequences where are the the the the consequences where are the the the the limits where where are the bugs the limits where where are the bugs the limits where where are the bugs the small and the big bugs that's the small and the big bugs that's the small and the big bugs that's the constant thing that the engineers are constant thing that the engineers are constant thing that the engineers are worried about so like I think Super

worried about so like I think Super worried about so like I think Super alignment is not quite the alignment is not quite the alignment is not quite the same as the um the kind of thing I'm same as the um the kind of thing I'm same as the um the kind of thing I'm referring to what Engineers are worried referring to what Engineers are worried referring to what Engineers are worried about super alignment is about super alignment is about super alignment is saying for future systems that we don't saying for future systems that we don't saying for future systems that we don't quite yet have how do we keep them safe quite yet have how do we keep them safe quite yet have how do we keep them safe if you're trying to be a step ahead it's if you're trying to be a step ahead it's if you're trying to be a step ahead it's it's a it's a different kind of problem it's a it's a different kind of problem it's a it's a different kind of problem because it's almost more philosophical because it's almost more philosophical because it's almost more philosophical it's a really tricky one because like it's a really tricky one because like it's a really tricky one because like you're you're trying you're trying to you're you're trying you're trying to you're you're trying you're trying to make prevent future systems from from make prevent future systems from from make prevent future systems from from escaping control of humans that's really escaping control of humans that's really escaping control of humans that's really I don't think there's I don't think there's I don't think there's been man is there anything akin to it in been man is there anything akin to it in been man is there anything akin to it in the history of humanity I don't think so the history of humanity I don't think so the history of humanity I don't think so right climate change but there there's a right climate change but there there's a right climate change but there there's a entire system which is climate which is entire system which is climate which is entire system which is climate which is incredibly complex which we don't have incredibly complex which we don't have incredibly complex which we don't have we have we have we have only tiny control of right it's its own only tiny control of right it's its own only tiny control of right it's its own system in this case we're building the system in this case we're building the system in this case we're building the system MH and so I how do you keep that system MH and so I how do you keep that system MH and so I how do you keep that system from becoming destructive that's system from becoming destructive that's system from becoming destructive that's a really diffic different problem than a really diffic different problem than a really diffic different problem than the current meetings that companies are the current meetings that companies are the current meetings that companies are having where the engineers are saying having where the engineers are saying having where the engineers are saying okay what like how powerful is this okay what like how powerful is this okay what like how powerful is this thing How does it go wrong thing How does it go wrong thing How does it go wrong um and as we train GPT 5 and train up um and as we train GPT 5 and train up um and as we train GPT 5 and train up future systems like where are the ways future systems like where are the ways future systems like where are the ways that can go wrong don't you think all that can go wrong don't you think all that can go wrong don't you think all those Engineers are constantly worrying those Engineers are constantly worrying those Engineers are constantly worrying about this thinking about this which is about this thinking about this which is about this thinking about this which is a little bit different than the super

a little bit different than the super a little bit different than the super alignment team that's thinking a little alignment team that's thinking a little alignment team that's thinking a little bit further into the future well I I bit further into the future well I I bit further into the future well I I think a lot think a lot think a lot of people who historically worked on of people who historically worked on of people who historically worked on AI never considered what happens when AI never considered what happens when AI never considered what happens when they succeed seart Russell speaks they succeed seart Russell speaks they succeed seart Russell speaks beautifully about that um let's look beautifully about that um let's look beautifully about that um let's look okay maybe super intelligence is too okay maybe super intelligence is too okay maybe super intelligence is too futuristic we can develop practical futuristic we can develop practical futuristic we can develop practical tools for it let's look at software tools for it let's look at software tools for it let's look at software today what is the state of Safety and today what is the state of Safety and today what is the state of Safety and Security of our user software things we Security of our user software things we Security of our user software things we give to millions of people there is no give to millions of people there is no give to millions of people there is no liability you click I agree what are you liability you click I agree what are you liability you click I agree what are you agreeing to nobody knows nobody is but agreeing to nobody knows nobody is but agreeing to nobody knows nobody is but you're basically saying it will spy on you're basically saying it will spy on you're basically saying it will spy on you corrupt your data kill your you corrupt your data kill your you corrupt your data kill your firstborn and you agree and you're not firstborn and you agree and you're not firstborn and you agree and you're not to the company that's the best they can to the company that's the best they can to the company that's the best they can do for mundane software word processor do for mundane software word processor do for mundane software word processor text software no liability no text software no liability no text software no liability no responsibility just as long as you agree responsibility just as long as you agree responsibility just as long as you agree not to Su us you can use it if this is a not to Su us you can use it if this is a not to Su us you can use it if this is a state-ofthe-art in systems which are state-ofthe-art in systems which are state-ofthe-art in systems which are narrow accountants stable manipulators narrow accountants stable manipulators narrow accountants stable manipulators why do we think we can do so much better why do we think we can do so much better why do we think we can do so much better with much more complex systems cross with much more complex systems cross with much more complex systems cross multiple domains in the environment with multiple domains in the environment with multiple domains in the environment with malevolent actors with again malevolent actors with again malevolent actors with again self-improvement with capabilities self-improvement with capabilities self-improvement with capabilities exceeding those of humans thinking about exceeding those of humans thinking about exceeding those of humans thinking about it I mean the liability thing is more it I mean the liability thing is more it I mean the liability thing is more about lawyers than killing firstborns about lawyers than killing firstborns about lawyers than killing firstborns but if clippy actually uh killed the but if clippy actually uh killed the but if clippy actually uh killed the child I think lawyers aside it would end

child I think lawyers aside it would end child I think lawyers aside it would end clippy and the company that owns clippy and the company that owns clippy and the company that owns clippy all right so it's not so much clippy all right so it's not so much clippy all right so it's not so much about there is there's two points to be about there is there's two points to be about there is there's two points to be made one is like man current software made one is like man current software made one is like man current software systems are are full of systems are are full of systems are are full of bugs and they could do a lot of damage bugs and they could do a lot of damage bugs and they could do a lot of damage and we don't know what kind it's they're and we don't know what kind it's they're and we don't know what kind it's they're unpredictable there's so much damage unpredictable there's so much damage unpredictable there's so much damage they could possibly do and then we kind they could possibly do and then we kind they could possibly do and then we kind of live in this uh Blissful illusion of live in this uh Blissful illusion of live in this uh Blissful illusion that everything is great and perfect and that everything is great and perfect and that everything is great and perfect and it it it works it's nevertheless it still somehow works it's nevertheless it still somehow works it's nevertheless it still somehow Works in many domains we see car Works in many domains we see car Works in many domains we see car manufacturing drug development the manufacturing drug development the manufacturing drug development the burden of proof is on a manufacturer of burden of proof is on a manufacturer of burden of proof is on a manufacturer of product or service to show their product product or service to show their product product or service to show their product or services safe or services safe or services safe it is not up to the user to prove that it is not up to the user to prove that it is not up to the user to prove that there are problems they have to do there are problems they have to do there are problems they have to do appropriate safety studies they have to appropriate safety studies they have to appropriate safety studies they have to get government approval for selling the get government approval for selling the get government approval for selling the product and they are still fully product and they are still fully product and they are still fully responsible for what happens we don't responsible for what happens we don't responsible for what happens we don't see any of that here they can deploy see any of that here they can deploy see any of that here they can deploy whatever they want and I have to explain whatever they want and I have to explain whatever they want and I have to explain how that system is going to kill how that system is going to kill how that system is going to kill everyone I I don't work for that company everyone I I don't work for that company everyone I I don't work for that company you have to explain to me how it's you have to explain to me how it's you have to explain to me how it's definitely cannot mess up that's because definitely cannot mess up that's because definitely cannot mess up that's because it's the very early days of such a it's the very early days of such a it's the very early days of such a technology government regulation is technology government regulation is technology government regulation is lagging behind they're really not lagging behind they're really not lagging behind they're really not techsavvy a regulation of any kind of techsavvy a regulation of any kind of techsavvy a regulation of any kind of software if if you look at like Congress software if if you look at like Congress software if if you look at like Congress talking about social media whenever Mark talking about social media whenever Mark talking about social media whenever Mark Zuckerberg and other CEOs show up the Zuckerberg and other CEOs show up the Zuckerberg and other CEOs show up the cluelessness that that uh Congress has cluelessness that that uh Congress has cluelessness that that uh Congress has about how technology works is is

about how technology works is is about how technology works is is incredible it's it's uh heartbreaking I incredible it's it's uh heartbreaking I incredible it's it's uh heartbreaking I agree completely but that's what scares agree completely but that's what scares agree completely but that's what scares me the responses when they start to get me the responses when they start to get me the responses when they start to get dangerous we'll really get it together dangerous we'll really get it together dangerous we'll really get it together the politicians will pass the right laws the politicians will pass the right laws the politicians will pass the right laws Engineers will solve the right problems Engineers will solve the right problems Engineers will solve the right problems we are not that good at many of those we are not that good at many of those we are not that good at many of those things we take forever and uh we are not things we take forever and uh we are not things we take forever and uh we are not early we are two years away according to early we are two years away according to early we are two years away according to prediction markets this is not a bias prediction markets this is not a bias prediction markets this is not a bias CEO fundraising this is what smartest CEO fundraising this is what smartest CEO fundraising this is what smartest people super forecasters are thinking of people super forecasters are thinking of people super forecasters are thinking of this this this problem I problem I problem I don't I'd like to push back about those don't I'd like to push back about those don't I'd like to push back about those pred I wonder what those prediction pred I wonder what those prediction pred I wonder what those prediction markets are about how they Define AGI markets are about how they Define AGI markets are about how they Define AGI that's wow to me and I want to know what that's wow to me and I want to know what that's wow to me and I want to know what they said about autonomous vehicles cuz they said about autonomous vehicles cuz they said about autonomous vehicles cuz I've heard a lot of experts and I've heard a lot of experts and I've heard a lot of experts and financial experts talk about autonomous financial experts talk about autonomous financial experts talk about autonomous vehicles and how it's going to be a vehicles and how it's going to be a vehicles and how it's going to be a multi-trillion dollar industry and all multi-trillion dollar industry and all multi-trillion dollar industry and all this kind of stuff and it's this kind of stuff and it's this kind of stuff and it's uh it's a small fund but if you have uh it's a small fund but if you have uh it's a small fund but if you have good Vision maybe you can zoom in on good Vision maybe you can zoom in on good Vision maybe you can zoom in on that and see the prediction dates that and see the prediction dates that and see the prediction dates descrition I have a large one if you're descrition I have a large one if you're descrition I have a large one if you're interested but I guess my fundamental interested but I guess my fundamental interested but I guess my fundamental question is how often they they write question is how often they they write question is how often they they write about technology about technology about technology I I I I I I I I I definitely there studies on their definitely there studies on their definitely there studies on their accuracy rates and all that you can look accuracy rates and all that you can look accuracy rates and all that you can look it up but even if they're wrong I'm just it up but even if they're wrong I'm just it up but even if they're wrong I'm just saying this is right now the best we saying this is right now the best we saying this is right now the best we have this is what Humanity came up with have this is what Humanity came up with have this is what Humanity came up with as the predicted date but again what as the predicted date but again what as the predicted date but again what they mean by AGI is really important they mean by AGI is really important they mean by AGI is really important there because there's uh the non-agent

there because there's uh the non-agent there because there's uh the non-agent like AGI and then there's the agent like like AGI and then there's the agent like like AGI and then there's the agent like AGI and I don't think it's as trivial as AGI and I don't think it's as trivial as AGI and I don't think it's as trivial as a rapper putting a wrap around uh a rapper putting a wrap around uh a rapper putting a wrap around uh uh one has lipstick and all it takes is uh one has lipstick and all it takes is uh one has lipstick and all it takes is to remove the lipstick I don't think to remove the lipstick I don't think to remove the lipstick I don't think it's that true you you may be completely it's that true you you may be completely it's that true you you may be completely right but what probability would you right but what probability would you right but what probability would you assign it you may be 10% wrong but we're assign it you may be 10% wrong but we're assign it you may be 10% wrong but we're betting all of Humanity on this betting all of Humanity on this betting all of Humanity on this distribution it seems irrational yeah distribution it seems irrational yeah distribution it seems irrational yeah it's definitely not like one or 0% yeah it's definitely not like one or 0% yeah it's definitely not like one or 0% yeah what are your thoughts by the way about what are your thoughts by the way about what are your thoughts by the way about current current current systems where they stand so GPT systems where they stand so GPT systems where they stand so GPT 40 claw 3 Gro 40 claw 3 Gro 40 claw 3 Gro Gemini we're like uh on the path to Gemini we're like uh on the path to Gemini we're like uh on the path to Super Super Super intelligence to agent like super intelligence to agent like super intelligence to agent like super intelligence where are intelligence where are intelligence where are we I think they all about the same we I think they all about the same we I think they all about the same obviously there are nuanced differences obviously there are nuanced differences obviously there are nuanced differences but in terms of capability I don't see a but in terms of capability I don't see a but in terms of capability I don't see a huge difference between them as I said huge difference between them as I said huge difference between them as I said in my opinion across all possible tasks in my opinion across all possible tasks in my opinion across all possible tasks they exceed performance of an average they exceed performance of an average they exceed performance of an average person yeah I think they starting to be person yeah I think they starting to be person yeah I think they starting to be better than an average Master student better than an average Master student better than an average Master student student at my student at my student at my University but uh they still have very University but uh they still have very University but uh they still have very big limitations if the next model is as big limitations if the next model is as big limitations if the next model is as improved as GPT 4 versus gpt3 we may see improved as GPT 4 versus gpt3 we may see improved as GPT 4 versus gpt3 we may see something very very very capable what do something very very very capable what do something very very very capable what do you feel about all this I mean you've you feel about all this I mean you've you feel about all this I mean you've been uh thinking about AI safety for a been uh thinking about AI safety for a been uh thinking about AI safety for a long long long long long long time and at least for

time and at least for time and at least for me the leaps I mean it probably started with Alpha zero with Alpha zero was mind-blowing for me and then the was mind-blowing for me and then the was mind-blowing for me and then the breakthroughs with l&m's even dpd2 but breakthroughs with l&m's even dpd2 but breakthroughs with l&m's even dpd2 but like just the the breakthroughs on llms like just the the breakthroughs on llms like just the the breakthroughs on llms just mind-blowing to me what does it just mind-blowing to me what does it just mind-blowing to me what does it feel like to be living in this day and feel like to be living in this day and feel like to be living in this day and age where all this talk about AGI feels age where all this talk about AGI feels age where all this talk about AGI feels like it like this is it actually might like it like this is it actually might like it like this is it actually might happen and quite soon meaning within our happen and quite soon meaning within our happen and quite soon meaning within our lifetime what what does it feel like so lifetime what what does it feel like so lifetime what what does it feel like so when I started working on this it was when I started working on this it was when I started working on this it was pure science fiction there was no pure science fiction there was no pure science fiction there was no funding no journals no conferences no funding no journals no conferences no funding no journals no conferences no one in Academia would dare to touch one in Academia would dare to touch one in Academia would dare to touch anything with the word singularity in it anything with the word singularity in it anything with the word singularity in it and I was Pretender at the time so I was and I was Pretender at the time so I was and I was Pretender at the time so I was pretty dumb um now you see touring Award pretty dumb um now you see touring Award pretty dumb um now you see touring Award winners publishing in science about how winners publishing in science about how winners publishing in science about how far behind we are according to them in far behind we are according to them in far behind we are according to them in addressing this problem so it's addressing this problem so it's addressing this problem so it's definitely a change it's uh difficult to definitely a change it's uh difficult to definitely a change it's uh difficult to keep up I used to be able to read every keep up I used to be able to read every keep up I used to be able to read every paper on AI safety then I was able to paper on AI safety then I was able to paper on AI safety then I was able to read the the best ones then the titles read the the best ones then the titles read the the best ones then the titles and now I don't even know what's going and now I don't even know what's going and now I don't even know what's going on by the time this interview is over on by the time this interview is over on by the time this interview is over they probably had GPT 6 released and I they probably had GPT 6 released and I they probably had GPT 6 released and I have to deal with that when I get back have to deal with that when I get back have to deal with that when I get back home so it's interesting yes there is home so it's interesting yes there is home so it's interesting yes there is now more opportunities I get invited to now more opportunities I get invited to now more opportunities I get invited to speak to smart people by the way I would speak to smart people by the way I would speak to smart people by the way I would have talked to you before any of have talked to you before any of have talked to you before any of this this is not like some trend of to

this this is not like some trend of to this this is not like some trend of to me it's we're still far away so just to me it's we're still far away so just to me it's we're still far away so just to be clear we're still far away from AGI be clear we're still far away from AGI be clear we're still far away from AGI but not far away in the but not far away in the but not far away in the sense relative to the magnitude of sense relative to the magnitude of sense relative to the magnitude of impact it can have we're not far away impact it can have we're not far away impact it can have we're not far away and we weren't far away 20 years ago and we weren't far away 20 years ago and we weren't far away 20 years ago because the impact a jack can have is on because the impact a jack can have is on because the impact a jack can have is on a scale of centuries it can end human a scale of centuries it can end human a scale of centuries it can end human civilization or it can transform it so civilization or it can transform it so civilization or it can transform it so like this discussion about one or two like this discussion about one or two like this discussion about one or two years versus one or two decades or even years versus one or two decades or even years versus one or two decades or even a 100 years not as important to me a 100 years not as important to me a 100 years not as important to me because it we're headed there this is because it we're headed there this is because it we're headed there this is like a like a like a human civilization scale question so U human civilization scale question so U human civilization scale question so U this is not just a Hot Topic is the most this is not just a Hot Topic is the most this is not just a Hot Topic is the most important problem we'll ever face it is important problem we'll ever face it is important problem we'll ever face it is not like anything we had to deal with not like anything we had to deal with not like anything we had to deal with before we never had birth of the NAA before we never had birth of the NAA before we never had birth of the NAA intelligence like aliens never visited intelligence like aliens never visited intelligence like aliens never visited us as far as I know so similar type of us as far as I know so similar type of us as far as I know so similar type of Problem by the way if an intelligent Problem by the way if an intelligent Problem by the way if an intelligent alien civilization visited us that's a alien civilization visited us that's a alien civilization visited us that's a similar kind of situation in some ways similar kind of situation in some ways similar kind of situation in some ways if you look at history anytime a more if you look at history anytime a more if you look at history anytime a more technologically advanced civilization technologically advanced civilization technologically advanced civilization visited a more primitive one the results visited a more primitive one the results visited a more primitive one the results were genocide every single time and were genocide every single time and were genocide every single time and sometimes the genocide is worse than sometimes the genocide is worse than sometimes the genocide is worse than other sometimes there's less suffering other sometimes there's less suffering other sometimes there's less suffering and more and more and more suffering and they always wondered but suffering and they always wondered but suffering and they always wondered but how can they kill us with those fire how can they kill us with those fire how can they kill us with those fire sticks and biological blankets and I sticks and biological blankets and I sticks and biological blankets and I mean Jenis Khan was nicer he offered the mean Jenis Khan was nicer he offered the mean Jenis Khan was nicer he offered the choice of join or or die but join

choice of join or or die but join choice of join or or die but join implies you have something to contribute implies you have something to contribute implies you have something to contribute what are you contributing to Super what are you contributing to Super what are you contributing to Super intelligence well in the intelligence well in the intelligence well in the zoo we're entertaining to zoo we're entertaining to zoo we're entertaining to watch to All Humans you know I just watch to All Humans you know I just watch to All Humans you know I just spent some time in the Amazon I watched spent some time in the Amazon I watched spent some time in the Amazon I watched ants for a long time and ants are kind ants for a long time and ants are kind ants for a long time and ants are kind of fascinating to watch I can watch them of fascinating to watch I can watch them of fascinating to watch I can watch them for a long time I'm sure there's a lot for a long time I'm sure there's a lot for a long time I'm sure there's a lot of value in watching humans cuz we're of value in watching humans cuz we're of value in watching humans cuz we're like um the interesting thing about like um the interesting thing about like um the interesting thing about humans you know like when you have a humans you know like when you have a humans you know like when you have a video game that's really well video game that's really well video game that's really well balanced because of the whole balanced because of the whole balanced because of the whole evolutionary process we've created the evolutionary process we've created the evolutionary process we've created the society as pretty well balanced like our society as pretty well balanced like our society as pretty well balanced like our our limitations as as humans and our our limitations as as humans and our our limitations as as humans and our capabilities are a balance from a video capabilities are a balance from a video capabilities are a balance from a video game perspective so we have wars we have game perspective so we have wars we have game perspective so we have wars we have conflicts we have cooperation like in a conflicts we have cooperation like in a conflicts we have cooperation like in a game theoretic way it's an interesting game theoretic way it's an interesting game theoretic way it's an interesting system to watch in the same way then system to watch in the same way then system to watch in the same way then andt colony is an interesting system to andt colony is an interesting system to andt colony is an interesting system to watch so like if I was an alien watch so like if I was an alien watch so like if I was an alien civilization I wouldn't want to disturb civilization I wouldn't want to disturb civilization I wouldn't want to disturb it I'd just watch it be interesting it I'd just watch it be interesting it I'd just watch it be interesting maybe perturb it every once in a while maybe perturb it every once in a while maybe perturb it every once in a while in interesting ways well we getting back in interesting ways well we getting back in interesting ways well we getting back to our simulation discussion from before to our simulation discussion from before to our simulation discussion from before how did it happen that we exist at how did it happen that we exist at how did it happen that we exist at exactly like the most interesting 20 30 exactly like the most interesting 20 30 exactly like the most interesting 20 30 years in a history of this civilization years in a history of this civilization years in a history of this civilization it's been around for 15 billion years it's been around for 15 billion years it's been around for 15 billion years yeah and that here we are what's the yeah and that here we are what's the yeah and that here we are what's the probability that we live in a simulation probability that we live in a simulation probability that we live in a simulation I know never to say 100% but pretty I know never to say 100% but pretty I know never to say 100% but pretty close to close to close to that is it possible to escape the that is it possible to escape the that is it possible to escape the simulation I have a paper about that simulation I have a paper about that simulation I have a paper about that this is just a first page teaser but this is just a first page teaser but this is just a first page teaser but it's like a nice 30 page document I'm

it's like a nice 30 page document I'm it's like a nice 30 page document I'm still here but uh yes how to hack the still here but uh yes how to hack the still here but uh yes how to hack the simulation is the title I spend a lot of simulation is the title I spend a lot of simulation is the title I spend a lot of time thinking about that that would be time thinking about that that would be time thinking about that that would be something I would want super something I would want super something I would want super intelligence to help us with and that's intelligence to help us with and that's intelligence to help us with and that's exactly what the paper is about we used exactly what the paper is about we used exactly what the paper is about we used AI boxing as a possible tool for control AI boxing as a possible tool for control AI boxing as a possible tool for control AI we realized AI will always Escape but AI we realized AI will always Escape but AI we realized AI will always Escape but that is a skill we might use to help us that is a skill we might use to help us that is a skill we might use to help us escape from our virtual box if we are in escape from our virtual box if we are in escape from our virtual box if we are in one yeah that you you have a lot of one yeah that you you have a lot of one yeah that you you have a lot of really great quotes here cluding Elam really great quotes here cluding Elam really great quotes here cluding Elam mus saying what's outside the simulation mus saying what's outside the simulation mus saying what's outside the simulation a question I asked him what he would ask a question I asked him what he would ask a question I asked him what he would ask an AGI system and he said he would ask an AGI system and he said he would ask an AGI system and he said he would ask what's outside the simulation that's a what's outside the simulation that's a what's outside the simulation that's a really good question to ask and maybe really good question to ask and maybe really good question to ask and maybe the followup is the title of the paper the followup is the title of the paper the followup is the title of the paper is how to how to get out or how to hack is how to how to get out or how to hack is how to how to get out or how to hack it the abstract reads many researchers it the abstract reads many researchers it the abstract reads many researchers have conjectured that the humankind is have conjectured that the humankind is have conjectured that the humankind is simulated along with the rest of the simulated along with the rest of the simulated along with the rest of the physical Universe in this paper we do physical Universe in this paper we do physical Universe in this paper we do not evaluate evidence for or against not evaluate evidence for or against not evaluate evidence for or against such a claim but instead ask a computer such a claim but instead ask a computer such a claim but instead ask a computer science question namely can we hack it science question namely can we hack it science question namely can we hack it more formally the question could be more formally the question could be more formally the question could be phrased as could generally intelligent phrased as could generally intelligent phrased as could generally intelligent agents placed in Virtual environments agents placed in Virtual environments agents placed in Virtual environments find a way to jailbreak out of them find a way to jailbreak out of them find a way to jailbreak out of them that's a fascinating question at a small that's a fascinating question at a small that's a fascinating question at a small scale like you can actually just scale like you can actually just scale like you can actually just construct experiments experiments okay can they how can they so a lot okay can they how can they so a lot okay can they how can they so a lot depends on intelligence of simulators depends on intelligence of simulators depends on intelligence of simulators right with uh humans boxing super

right with uh humans boxing super right with uh humans boxing super intelligence the entity in a box was intelligence the entity in a box was intelligence the entity in a box was smarter than us presumed to be if the smarter than us presumed to be if the smarter than us presumed to be if the simulators are much smarter than us and simulators are much smarter than us and simulators are much smarter than us and the super intelligence we create then the super intelligence we create then the super intelligence we create then probably they can contain us because probably they can contain us because probably they can contain us because greater intelligence can control lower greater intelligence can control lower greater intelligence can control lower intelligence at least for some time on intelligence at least for some time on intelligence at least for some time on the other hand if our super intelligence the other hand if our super intelligence the other hand if our super intelligence somehow for whatever reason despite somehow for whatever reason despite somehow for whatever reason despite having only local resources manages to having only local resources manages to having only local resources manages to fo levels Beyond it maybe it will fo levels Beyond it maybe it will fo levels Beyond it maybe it will succeed maybe the security is not that succeed maybe the security is not that succeed maybe the security is not that important to them maybe it's important to them maybe it's important to them maybe it's entertainment system so there is no entertainment system so there is no entertainment system so there is no security and it's easy to hack it if I security and it's easy to hack it if I security and it's easy to hack it if I was creating a was creating a was creating a simulation I would want the possibility simulation I would want the possibility simulation I would want the possibility to escape it to be there so the to escape it to be there so the to escape it to be there so the possibility of f of a of a takeoff where possibility of f of a of a takeoff where possibility of f of a of a takeoff where the agents become smart enough to escape the agents become smart enough to escape the agents become smart enough to escape the simulation would be the thing i' be the simulation would be the thing i' be the simulation would be the thing i' be waiting for that could be the test waiting for that could be the test waiting for that could be the test you're actually performing are you smart you're actually performing are you smart you're actually performing are you smart enough to escape your puzzle enough to escape your puzzle enough to escape your puzzle that could be like first of all first of that could be like first of all first of that could be like first of all first of all we mentioned touring test that is a all we mentioned touring test that is a all we mentioned touring test that is a good test are you smart enough like this good test are you smart enough like this good test are you smart enough like this is a game do a realize this world is not is a game do a realize this world is not is a game do a realize this world is not real it's just a test that's a really real it's just a test that's a really real it's just a test that's a really good good good test that's a really good test that's a test that's a really good test that's a test that's a really good test that's a really good test even for AI systems now really good test even for AI systems now really good test even for AI systems now like can can we construct the simulated like can can we construct the simulated like can can we construct the simulated world for world for world for them and them and them and can they realize that they are inside

can they realize that they are inside can they realize that they are inside that world and Escape that world and Escape that world and Escape it have you have you played around have it have you have you played around have it have you have you played around have you seen anybody play around with like you seen anybody play around with like you seen anybody play around with like rigorously constructing such experiments rigorously constructing such experiments rigorously constructing such experiments not specifically escaping for agents but not specifically escaping for agents but not specifically escaping for agents but a lot of testing is done in Virtual a lot of testing is done in Virtual a lot of testing is done in Virtual Worlds I think there is a quote the Worlds I think there is a quote the Worlds I think there is a quote the first one maybe which kind of talks first one maybe which kind of talks first one maybe which kind of talks about I realizing but not humans is that about I realizing but not humans is that about I realizing but not humans is that I'm reading upside I'm reading upside I'm reading upside down yeah this one few so the and the down yeah this one few so the and the down yeah this one few so the and the first quote is from Swift on security first quote is from Swift on security first quote is from Swift on security let me out the artificial intelligence let me out the artificial intelligence let me out the artificial intelligence yelled aimlessly into walls themselves yelled aimlessly into walls themselves yelled aimlessly into walls themselves pacing the room out of what the engineer pacing the room out of what the engineer pacing the room out of what the engineer asked the simulation you have me in but asked the simulation you have me in but asked the simulation you have me in but we're in the real world the machine we're in the real world the machine we're in the real world the machine paused and shuddered for its captors oh paused and shuddered for its captors oh paused and shuddered for its captors oh God you can't tell yeah that's a big God you can't tell yeah that's a big God you can't tell yeah that's a big leap to take for a system to realize leap to take for a system to realize leap to take for a system to realize that there there's a box and you're that there there's a box and you're that there there's a box and you're inside it I wonder if like a language model can it I wonder if like a language model can do that they're smart enough to talk do that they're smart enough to talk do that they're smart enough to talk about those Concepts I had many good about those Concepts I had many good about those Concepts I had many good philosophical discussions about such philosophical discussions about such philosophical discussions about such issues they issues they issues they usually at least as interesting as most usually at least as interesting as most usually at least as interesting as most humans in humans in humans in that what do you think about AI safety that what do you think about AI safety that what do you think about AI safety in the simulated world so can you can in the simulated world so can you can in the simulated world so can you can you have kind you have kind you have kind of create simulated worlds where you can

of create simulated worlds where you can of create simulated worlds where you can test play with a dangerous AGI system test play with a dangerous AGI system test play with a dangerous AGI system yeah and that was exactly what one of yeah and that was exactly what one of yeah and that was exactly what one of the early papers was on AI boxing how to the early papers was on AI boxing how to the early papers was on AI boxing how to leak proof leak proof leak proof Singularity uh if they're smart enough Singularity uh if they're smart enough Singularity uh if they're smart enough to realize they in a simulation they'll to realize they in a simulation they'll to realize they in a simulation they'll act appropriately until you let them act appropriately until you let them act appropriately until you let them out if they can hack out they will and out if they can hack out they will and out if they can hack out they will and if you're observing them that means if you're observing them that means if you're observing them that means there is a communication Channel and there is a communication Channel and there is a communication Channel and that's enough for social engineering that's enough for social engineering that's enough for social engineering attack so really it's uh it's impossible attack so really it's uh it's impossible attack so really it's uh it's impossible to test an AGI system that's dangerous to test an AGI system that's dangerous to test an AGI system that's dangerous enough to destroy Humanity because it's enough to destroy Humanity because it's enough to destroy Humanity because it's either going to what escape the either going to what escape the either going to what escape the simulation or pretend it's safe until simulation or pretend it's safe until simulation or pretend it's safe until it's let out either either or can force it's let out either either or can force it's let out either either or can force you to let it out blackmail you bribe you to let it out blackmail you bribe you to let it out blackmail you bribe you promise you infinite life commed to you promise you infinite life commed to you promise you infinite life commed to virgins whatever yeah it can be virgins whatever yeah it can be virgins whatever yeah it can be convincing convincing convincing charismatic the social engineering is charismatic the social engineering is charismatic the social engineering is really scary to me cuz it feels like really scary to me cuz it feels like really scary to me cuz it feels like humans humans humans are very uh are very uh are very uh engineerable like we're lonely we're engineerable like we're lonely we're engineerable like we're lonely we're flawed we're flawed we're flawed we're Moody and it feels like a AI system with Moody and it feels like a AI system with Moody and it feels like a AI system with a with a nice voice can convince us to a with a nice voice can convince us to a with a nice voice can convince us to do basically do basically do basically anything at at an extremely large scale it's also possible that the in the

scale it's also possible that the in the uh increased proliferation of all this uh increased proliferation of all this uh increased proliferation of all this technology will force humans to uh get technology will force humans to uh get technology will force humans to uh get away from technology and value this like away from technology and value this like away from technology and value this like in-person in-person in-person communication basically don't trust communication basically don't trust communication basically don't trust anything anything anything else it's possible um surprisingly so at else it's possible um surprisingly so at else it's possible um surprisingly so at University I see huge growth in online University I see huge growth in online University I see huge growth in online courses and shrinkage of in person where courses and shrinkage of in person where courses and shrinkage of in person where I always understood in person being the I always understood in person being the I always understood in person being the only value I offer so it's only value I offer so it's only value I offer so it's puzzling I don't know that there could puzzling I don't know that there could puzzling I don't know that there could be a a trend towards the be a a trend towards the be a a trend towards the inperson because of deep fakes because inperson because of deep fakes because inperson because of deep fakes because of uh inability to of uh inability to of uh inability to trust in inability to trust the veracity trust in inability to trust the veracity trust in inability to trust the veracity of anything on the internet so the only of anything on the internet so the only of anything on the internet so the only way to verify is by being there in way to verify is by being there in way to verify is by being there in person but not yet person but not yet person but not yet uh why do you think aliens haven't come uh why do you think aliens haven't come uh why do you think aliens haven't come here yet so there is a lot of real here yet so there is a lot of real here yet so there is a lot of real estate out there it would be surprising estate out there it would be surprising estate out there it would be surprising if it was all for nothing it was empty if it was all for nothing it was empty if it was all for nothing it was empty and the moment there is Advanced enough and the moment there is Advanced enough and the moment there is Advanced enough biological civilization kind of self biological civilization kind of self biological civilization kind of self starting civilization it probably starts starting civilization it probably starts starting civilization it probably starts sending out the Norman probes everywhere sending out the Norman probes everywhere sending out the Norman probes everywhere and so for every biological one there and so for every biological one there and so for every biological one there got to be trillions of robot populated got to be trillions of robot populated got to be trillions of robot populated planets which probably do more of the planets which probably do more of the planets which probably do more of the same so it is uh uh likely same so it is uh uh likely same so it is uh uh likely statistically so now the fact that we statistically so now the fact that we statistically so now the fact that we haven't seen them one one answer is haven't seen them one one answer is haven't seen them one one answer is we're in a we're in a we're in a simulation it'd

simulation it'd simulation it'd be it would be hard to like add simulate be it would be hard to like add simulate be it would be hard to like add simulate or it be not interesting to simulate all or it be not interesting to simulate all or it be not interesting to simulate all those other intelligences it's a better those other intelligences it's a better those other intelligences it's a better it's better for the narrative you have it's better for the narrative you have it's better for the narrative you have to have a control variable yeah to have a control variable yeah to have a control variable yeah exactly exactly exactly okay uh but it's also possible that okay uh but it's also possible that okay uh but it's also possible that there is if we're not in simulation that there is if we're not in simulation that there is if we're not in simulation that there is a great filter that there is a great filter that there is a great filter that that naturally a lot of civilizations that naturally a lot of civilizations that naturally a lot of civilizations get to this point where there's super get to this point where there's super get to this point where there's super intelligent agents and then it just intelligent agents and then it just intelligent agents and then it just goes just dies so maybe uh throughout goes just dies so maybe uh throughout goes just dies so maybe uh throughout our galaxy and throughout the Universe our galaxy and throughout the Universe our galaxy and throughout the Universe there's just a bunch of dead alien there's just a bunch of dead alien there's just a bunch of dead alien civilizations it's possible I used to civilizations it's possible I used to civilizations it's possible I used to think that AI was the great filter but I think that AI was the great filter but I think that AI was the great filter but I would expect like a wall of comporium would expect like a wall of comporium would expect like a wall of comporium approaching us a speed of light or approaching us a speed of light or approaching us a speed of light or robots or something and I don't see it robots or something and I don't see it robots or something and I don't see it so it would still make a lot of noise it so it would still make a lot of noise it so it would still make a lot of noise it might not be interesting it might not might not be interesting it might not might not be interesting it might not possess Consciousness what we've been possess Consciousness what we've been possess Consciousness what we've been talking talking talking about it sounds like both you and I like about it sounds like both you and I like about it sounds like both you and I like humans some humans some humans some humans humans on the whole and so and we humans humans on the whole and so and we humans humans on the whole and so and we would like to preserve the the flame of would like to preserve the the flame of would like to preserve the the flame of human consciousness uh what do you think human consciousness uh what do you think human consciousness uh what do you think makes humans makes humans makes humans special that we would like to preserve special that we would like to preserve special that we would like to preserve them are we just being them are we just being them are we just being selfish or is there something special selfish or is there something special selfish or is there something special about humans so the only thing which about humans so the only thing which about humans so the only thing which matters is consciousness outside of it matters is consciousness outside of it matters is consciousness outside of it nothing else matters and internal states nothing else matters and internal states nothing else matters and internal states of qualia pain pleasure it seems that it

of qualia pain pleasure it seems that it of qualia pain pleasure it seems that it is unique to living beings I'm not aware is unique to living beings I'm not aware is unique to living beings I'm not aware of anyone claiming that I can torture a of anyone claiming that I can torture a of anyone claiming that I can torture a piece of software in a meaningful way piece of software in a meaningful way piece of software in a meaningful way there is a society for prevention of there is a society for prevention of there is a society for prevention of suffering to learning algorithms but uh suffering to learning algorithms but uh suffering to learning algorithms but uh a real a real a real thing many things are real on the thing many things are real on the thing many things are real on the internet but uh uh I I don't think internet but uh uh I I don't think internet but uh uh I I don't think anyone if I told them you know sit down anyone if I told them you know sit down anyone if I told them you know sit down and write a function to feel pain they and write a function to feel pain they and write a function to feel pain they would go beyond having an integer would go beyond having an integer would go beyond having an integer variable called pain and increasing the variable called pain and increasing the variable called pain and increasing the count so we don't know how to do it and count so we don't know how to do it and count so we don't know how to do it and that's that's that's unique uh that's what creates meaning it unique uh that's what creates meaning it unique uh that's what creates meaning it would be kind would be kind would be kind of as Bostrom calls a Disneyland without of as Bostrom calls a Disneyland without of as Bostrom calls a Disneyland without children if that was gone do you think children if that was gone do you think children if that was gone do you think Consciousness can be um engineered in Consciousness can be um engineered in Consciousness can be um engineered in artificial artificial artificial systems here let me uh let me go to 2011 systems here let me uh let me go to 2011 systems here let me uh let me go to 2011 paper that you wrote robot paper that you wrote robot paper that you wrote robot rights lastly we would like to address a rights lastly we would like to address a rights lastly we would like to address a subbranch of machine ethics which on the subbranch of machine ethics which on the subbranch of machine ethics which on the surface has little to do with safety but surface has little to do with safety but surface has little to do with safety but which is claimed to play a role in which is claimed to play a role in which is claimed to play a role in decision-making by ethical machines decision-making by ethical machines decision-making by ethical machines robot robot robot rights um so do do you think it's rights um so do do you think it's rights um so do do you think it's possible to engineer Consciousness in possible to engineer Consciousness in possible to engineer Consciousness in the machines and thereby the question the machines and thereby the question the machines and thereby the question extends to our legal system do you think extends to our legal system do you think extends to our legal system do you think uh at that point robot should have uh at that point robot should have uh at that point robot should have rights yeah I think I think we rights yeah I think I think we rights yeah I think I think we can I think it's possible to create can I think it's possible to create can I think it's possible to create Consciousness in machines I tried

Consciousness in machines I tried Consciousness in machines I tried designing a test for it with mixed designing a test for it with mixed designing a test for it with mixed success that paper talked about problems success that paper talked about problems success that paper talked about problems with with with giving uh civil rights to AI which can giving uh civil rights to AI which can giving uh civil rights to AI which can reproduce quickly and outvote humans reproduce quickly and outvote humans reproduce quickly and outvote humans essentially taking over a government essentially taking over a government essentially taking over a government system by simply voting for their system by simply voting for their system by simply voting for their controlled candidates as for um controlled candidates as for um controlled candidates as for um Consciousness in humans and other agents Consciousness in humans and other agents Consciousness in humans and other agents uh I have a paper where I propos relying uh I have a paper where I propos relying uh I have a paper where I propos relying on experience of optical illusions yeah on experience of optical illusions yeah on experience of optical illusions yeah if I can design a novel optical illusion if I can design a novel optical illusion if I can design a novel optical illusion and show it to an agent an alien a robot and show it to an agent an alien a robot and show it to an agent an alien a robot and they describe it exactly as I do and they describe it exactly as I do and they describe it exactly as I do it's very hard for me to argue that they it's very hard for me to argue that they it's very hard for me to argue that they haven't experienced that it's not part haven't experienced that it's not part haven't experienced that it's not part of a picture it's part of their software of a picture it's part of their software of a picture it's part of their software and Hardware representation a bug in and Hardware representation a bug in and Hardware representation a bug in their code which goes oh that triangle their code which goes oh that triangle their code which goes oh that triangle is is is rotating okay and I've been told it's rotating okay and I've been told it's rotating okay and I've been told it's really dumb and really brilliant by really dumb and really brilliant by really dumb and really brilliant by different philosophers so I am different philosophers so I am different philosophers so I am still so but now we finally have still so but now we finally have still so but now we finally have technology to test it we have tools we technology to test it we have tools we technology to test it we have tools we have AIS if someone wants to run this have AIS if someone wants to run this have AIS if someone wants to run this experiment I'm happy to collaborate so experiment I'm happy to collaborate so experiment I'm happy to collaborate so this is a test for Consciousness for this is a test for Consciousness for this is a test for Consciousness for internal state of experience that we internal state of experience that we internal state of experience that we share bugs it will show that we share share bugs it will show that we share share bugs it will show that we share common experiences if they have common experiences if they have common experiences if they have completely different internal States it completely different internal States it completely different internal States it would not register for us but it's a would not register for us but it's a would not register for us but it's a positive test if they pass it Time After positive test if they pass it Time After positive test if they pass it Time After Time with probability increasing for Time with probability increasing for Time with probability increasing for every multiple choice then you have no every multiple choice then you have no every multiple choice then you have no choice but to either accept that they choice but to either accept that they choice but to either accept that they have access to a conscious model or they have access to a conscious model or they have access to a conscious model or they are themselves so the reason Illusions

are themselves so the reason Illusions are themselves so the reason Illusions are interesting is I guess are interesting is I guess are interesting is I guess because it's a it's a really weird because it's a it's a really weird because it's a it's a really weird experience and if you both share that experience and if you both share that experience and if you both share that weird experience that's not there in the weird experience that's not there in the weird experience that's not there in the Bland physical description of the raw Bland physical description of the raw Bland physical description of the raw data that means that puts more emphasis on the actual that puts more emphasis on the actual experience and we know animals can experience and we know animals can experience and we know animals can experience some optical illusion so we experience some optical illusion so we experience some optical illusion so we know they have certain types of know they have certain types of know they have certain types of Consciousness as a result I would say Consciousness as a result I would say Consciousness as a result I would say yeah well that that just goes to my yeah well that that just goes to my yeah well that that just goes to my sense that the flaws and the bugs is sense that the flaws and the bugs is sense that the flaws and the bugs is what makes humans special makes living what makes humans special makes living what makes humans special makes living form special so you're saying like yeah form special so you're saying like yeah form special so you're saying like yeah a future not a bug it's a feure the bug a future not a bug it's a feure the bug a future not a bug it's a feure the bug is the feature who okay that's a that's is the feature who okay that's a that's is the feature who okay that's a that's a cool test for Consciousness and you a cool test for Consciousness and you a cool test for Consciousness and you think that can be engineered here then think that can be engineered here then think that can be engineered here then so there have to be novel Illusions if so there have to be novel Illusions if so there have to be novel Illusions if it can just Google the answer it's it can just Google the answer it's it can just Google the answer it's useless you have to come up with novel useless you have to come up with novel useless you have to come up with novel Illusions which we tried automating and Illusions which we tried automating and Illusions which we tried automating and failed so if someone can develop a failed so if someone can develop a failed so if someone can develop a system capable of producing novel system capable of producing novel system capable of producing novel optical illusions on demand then we can optical illusions on demand then we can optical illusions on demand then we can definitely administer that test on definitely administer that test on definitely administer that test on significant scale with good results significant scale with good results significant scale with good results first of all pretty cool idea um I don't first of all pretty cool idea um I don't first of all pretty cool idea um I don't know if it's a good General test of know if it's a good General test of know if it's a good General test of Consciousness it's a good component of Consciousness it's a good component of Consciousness it's a good component of that and no matter what it's just a cool that and no matter what it's just a cool that and no matter what it's just a cool idea so um put me in the camp of people idea so um put me in the camp of people idea so um put me in the camp of people that like it uh but you don't think like that like it uh but you don't think like that like it uh but you don't think like a touring test style Imitation Of a touring test style Imitation Of a touring test style Imitation Of Consciousness is a good test like if you Consciousness is a good test like if you Consciousness is a good test like if you can convince a lot of humans that you're can convince a lot of humans that you're can convince a lot of humans that you're conscious that doesn't that to you is

conscious that doesn't that to you is conscious that doesn't that to you is not impressive there is so much data in not impressive there is so much data in not impressive there is so much data in the internet I know exactly what to say the internet I know exactly what to say the internet I know exactly what to say then you ask me common human questions then you ask me common human questions then you ask me common human questions what does pain feel like what does what does pain feel like what does what does pain feel like what does pleasure feel like all that is pleasure feel like all that is pleasure feel like all that is googleable I I think to me Consciousness googleable I I think to me Consciousness googleable I I think to me Consciousness is closely tied to suffering so you can is closely tied to suffering so you can is closely tied to suffering so you can illust Your Capacity to suffer but with illust Your Capacity to suffer but with illust Your Capacity to suffer but with I guess with words there's so much data I guess with words there's so much data I guess with words there's so much data that you can say you can pretend your that you can say you can pretend your that you can say you can pretend your suffering and you can do so very suffering and you can do so very suffering and you can do so very convincingly there are simulators for convincingly there are simulators for convincingly there are simulators for Torture Games where the Avatar screams Torture Games where the Avatar screams Torture Games where the Avatar screams in pain begs to stop I mean that was a in pain begs to stop I mean that was a in pain begs to stop I mean that was a part of kind of standard psychology part of kind of standard psychology part of kind of standard psychology research you say it so uh calmly it research you say it so uh calmly it research you say it so uh calmly it sounds pretty dark uh welcome to sounds pretty dark uh welcome to sounds pretty dark uh welcome to humanity yeah uh yeah it's like a hitch haacker guide uh yeah it's like a hitch haacker guide summary mostly summary mostly summary mostly harmless I would I would love to get a harmless I would I would love to get a harmless I would I would love to get a good summary when all this is said and good summary when all this is said and good summary when all this is said and done when Earth is no longer a thing done when Earth is no longer a thing done when Earth is no longer a thing whatever a million a billion years from whatever a million a billion years from whatever a million a billion years from now like what's a good summary of what now like what's a good summary of what now like what's a good summary of what happened here it's happened here it's happened here it's interesting I think AI will play a big interesting I think AI will play a big interesting I think AI will play a big part of that part of that part of that summary and hopefully humans will too summary and hopefully humans will too summary and hopefully humans will too what do you think about the merger of what do you think about the merger of what do you think about the merger of the too so one of the things that Elon the too so one of the things that Elon the too so one of the things that Elon and yur Link talk about is one of the and yur Link talk about is one of the and yur Link talk about is one of the ways for us to achieve AI safety is ways for us to achieve AI safety is ways for us to achieve AI safety is to ride the wave of AGI so by merging so to ride the wave of AGI so by merging so to ride the wave of AGI so by merging so incredible technology in a narrow sense

incredible technology in a narrow sense incredible technology in a narrow sense to help the disabled just amazing to help the disabled just amazing to help the disabled just amazing support at support at support at 100% for long-term Hybrid models both 100% for long-term Hybrid models both 100% for long-term Hybrid models both parts need to contribute something to parts need to contribute something to parts need to contribute something to the overall system right now we are the overall system right now we are the overall system right now we are still more capable in many ways so still more capable in many ways so still more capable in many ways so having this connection to AI would be having this connection to AI would be having this connection to AI would be incredible would make me super human in incredible would make me super human in incredible would make me super human in many ways after a while if I'm no longer many ways after a while if I'm no longer many ways after a while if I'm no longer smarter more creative really don't smarter more creative really don't smarter more creative really don't contribute much the system finds me as a contribute much the system finds me as a contribute much the system finds me as a biological bottleneck and even biological bottleneck and even biological bottleneck and even explicitly or implicitly I'm removed explicitly or implicitly I'm removed explicitly or implicitly I'm removed from any participation in the system so from any participation in the system so from any participation in the system so it's like uh the appendix by the way the it's like uh the appendix by the way the it's like uh the appendix by the way the appendix is still around appendix is still around appendix is still around so even if it's you said so even if it's you said so even if it's you said bottleneck I don't know if we become a bottleneck I don't know if we become a bottleneck I don't know if we become a bottleneck we just might not have much bottleneck we just might not have much bottleneck we just might not have much use there a different thing than use there a different thing than use there a different thing than bottleneck wasting valuable energy by bottleneck wasting valuable energy by bottleneck wasting valuable energy by being there we don't waste that much being there we don't waste that much being there we don't waste that much energy we're pretty energy energy we're pretty energy energy we're pretty energy efficient we could just stick around efficient we could just stick around efficient we could just stick around like the appendix come on now that's the like the appendix come on now that's the like the appendix come on now that's the future of we all dream about become an future of we all dream about become an future of we all dream about become an appendix to the history book of appendix to the history book of appendix to the history book of humanity well and also the Consciousness humanity well and also the Consciousness humanity well and also the Consciousness thing the peculiar particular kind of thing the peculiar particular kind of thing the peculiar particular kind of Consciousness that humans have that Consciousness that humans have that Consciousness that humans have that might be useful that might be really might be useful that might be really might be useful that might be really hard to simulate hard to simulate hard to simulate but you you said that like how would but you you said that like how would but you you said that like how would that look like if you can engineer that that look like if you can engineer that that look like if you can engineer that in in Silicon Consciousness in in Silicon Consciousness in in Silicon Consciousness Consciousness I assume you are conscious Consciousness I assume you are conscious Consciousness I assume you are conscious I have no idea how to test for it or how I have no idea how to test for it or how I have no idea how to test for it or how it impacts you in any way whatsoever it impacts you in any way whatsoever it impacts you in any way whatsoever right now you can perfectly simulate all

right now you can perfectly simulate all right now you can perfectly simulate all of it without making any any different of it without making any any different of it without making any any different observations for me but to do it in a observations for me but to do it in a observations for me but to do it in a computer how would you do that cuz you computer how would you do that cuz you computer how would you do that cuz you kind of said that you think it's kind of said that you think it's kind of said that you think it's possible to do that so it may be an possible to do that so it may be an possible to do that so it may be an emergent phenomena we seem to get it U emergent phenomena we seem to get it U emergent phenomena we seem to get it U through evolutionary through evolutionary through evolutionary process uh it's not obvious how it helps process uh it's not obvious how it helps process uh it's not obvious how it helps us to survive us to survive us to survive better but uh maybe it's an internal better but uh maybe it's an internal better but uh maybe it's an internal kind of goey which allows us to better kind of goey which allows us to better kind of goey which allows us to better manipulate the world simplifies a lot of manipulate the world simplifies a lot of manipulate the world simplifies a lot of uh control uh control uh control structures uh that's one area where we structures uh that's one area where we structures uh that's one area where we have very very little progress lots of have very very little progress lots of have very very little progress lots of papers lots of research but papers lots of research but papers lots of research but Consciousness is not a big Consciousness is not a big Consciousness is not a big big area of successful Discovery so far big area of successful Discovery so far big area of successful Discovery so far a lot of people think that machines a lot of people think that machines a lot of people think that machines would have to be conscious to be would have to be conscious to be would have to be conscious to be dangerous that's a big dangerous that's a big dangerous that's a big misconception there is absolutely no misconception there is absolutely no misconception there is absolutely no need for this very powerful optimizing need for this very powerful optimizing need for this very powerful optimizing agent to feel anything while it's agent to feel anything while it's agent to feel anything while it's performing things on you but what do you performing things on you but what do you performing things on you but what do you think about this the the the whole think about this the the the whole think about this the the the whole science of emergence in general so I science of emergence in general so I science of emergence in general so I don't know how much you know about don't know how much you know about don't know how much you know about cellular autometa or these simplified cellular autometa or these simplified cellular autometa or these simplified systems where that stud this very systems where that stud this very systems where that stud this very question from Simple Rules emerges question from Simple Rules emerges question from Simple Rules emerges complexity I attended wol from summer complexity I attended wol from summer complexity I attended wol from summer school I love Steven very much I I love school I love Steven very much I I love school I love Steven very much I I love his work I love cell aoma so uh I just his work I love cell aoma so uh I just his work I love cell aoma so uh I just would love to get your thoughts how would love to get your thoughts how would love to get your thoughts how that fits into your view in the

that fits into your view in the that fits into your view in the emergence of intelligence in AGI systems emergence of intelligence in AGI systems emergence of intelligence in AGI systems and maybe just even simply what do you and maybe just even simply what do you and maybe just even simply what do you make of the fact that this complexity make of the fact that this complexity make of the fact that this complexity can emerge from such simple rules so can emerge from such simple rules so can emerge from such simple rules so very rule is simple but the size of a very rule is simple but the size of a very rule is simple but the size of a space is still huge and the neural space is still huge and the neural space is still huge and the neural networks were really the first discovery networks were really the first discovery networks were really the first discovery in AI 100 years ago the first papers in AI 100 years ago the first papers in AI 100 years ago the first papers were published on neural networks we were published on neural networks we were published on neural networks we just didn't have enough compute to make just didn't have enough compute to make just didn't have enough compute to make them work I can give you a rule such as them work I can give you a rule such as them work I can give you a rule such as start printing progressively larger start printing progressively larger start printing progressively larger strings that's it one sentence it will strings that's it one sentence it will strings that's it one sentence it will output everything every program every output everything every program every output everything every program every DNA code everything in that rule you DNA code everything in that rule you DNA code everything in that rule you need intelligence to to filter it out need intelligence to to filter it out need intelligence to to filter it out obviously to make it useful but simple obviously to make it useful but simple obviously to make it useful but simple generation is not that difficult and a generation is not that difficult and a generation is not that difficult and a lot of those systems uh end up being T lot of those systems uh end up being T lot of those systems uh end up being T in complete systems so they are in complete systems so they are in complete systems so they are Universal and we expect that level of Universal and we expect that level of Universal and we expect that level of complexity from them what I like about complexity from them what I like about complexity from them what I like about uh Wolf's work is that he talks about uh Wolf's work is that he talks about uh Wolf's work is that he talks about irreducibility you have to run the irreducibility you have to run the irreducibility you have to run the simulation you cannot predict what it's simulation you cannot predict what it's simulation you cannot predict what it's going to do ahead of time and I think going to do ahead of time and I think going to do ahead of time and I think that's very relevant to what we are that's very relevant to what we are that's very relevant to what we are talking about with those very complex talking about with those very complex talking about with those very complex systems until you live through it you systems until you live through it you systems until you live through it you cannot ahead of time tell me exactly cannot ahead of time tell me exactly cannot ahead of time tell me exactly what it's going to do irreducibility what it's going to do irreducibility what it's going to do irreducibility means that for a sufficiently complex means that for a sufficiently complex means that for a sufficiently complex system you have to run the thing you system you have to run the thing you system you have to run the thing you have to uh you can't predict what's have to uh you can't predict what's have to uh you can't predict what's going to happen in the universe you have going to happen in the universe you have going to happen in the universe you have to create a new universe and run the to create a new universe and run the to create a new universe and run the thing Big Bang the whole thing but thing Big Bang the whole thing but thing Big Bang the whole thing but running it may be consequential as well

running it may be consequential as well running it may be consequential as well it might destroy humans and to you there's no chance humans and to you there's no chance that AI somehow carry the flame of that AI somehow carry the flame of that AI somehow carry the flame of Consciousness the flame of specialness Consciousness the flame of specialness Consciousness the flame of specialness and awesomeness that is and awesomeness that is and awesomeness that is humans it may somehow but I still feel humans it may somehow but I still feel humans it may somehow but I still feel kind of bad that it killed all of us I kind of bad that it killed all of us I kind of bad that it killed all of us I would prefer that doesn't would prefer that doesn't would prefer that doesn't happen I can be happy for others but to happen I can be happy for others but to happen I can be happy for others but to a certain degree it would be nice if we a certain degree it would be nice if we a certain degree it would be nice if we stuck around for a long time at least stuck around for a long time at least stuck around for a long time at least give us a planet the human Planet it'd give us a planet the human Planet it'd give us a planet the human Planet it'd be nice for it to be Earth and then they be nice for it to be Earth and then they be nice for it to be Earth and then they can go elsewhere since they're so smart can go elsewhere since they're so smart can go elsewhere since they're so smart they can colonize Mars do you think they they could uh Mars do you think they they could uh help convert US to uh you know type one help convert US to uh you know type one help convert US to uh you know type one type two type three let's just take the type two type three let's just take the type two type three let's just take the type type type two uh civilization on the CF scale like two uh civilization on the CF scale like two uh civilization on the CF scale like help us help us help us help us help us help us humans expand out into the cosmos so all humans expand out into the cosmos so all humans expand out into the cosmos so all of it goes back to are we somehow of it goes back to are we somehow of it goes back to are we somehow controlling it are we getting results we controlling it are we getting results we controlling it are we getting results we want if yes then everything's possible want if yes then everything's possible want if yes then everything's possible yes they can definitely help us with yes they can definitely help us with yes they can definitely help us with science engineering science engineering science engineering exploration uh in every way conceivable exploration uh in every way conceivable exploration uh in every way conceivable but it's a big if this whole thing about but it's a big if this whole thing about but it's a big if this whole thing about control though humans are bad with control though humans are bad with control though humans are bad with control because the moment they gain control because the moment they gain control because the moment they gain control they be they can also easily control they be they can also easily control they be they can also easily become too become too become too controlling it's the whole the more controlling it's the whole the more controlling it's the whole the more control you have the more you want it

control you have the more you want it control you have the more you want it it's the the old power corrupts and it's the the old power corrupts and it's the the old power corrupts and absolute power corrupts absolute power corrupts absolute power corrupts absolutely and it feels like control absolutely and it feels like control absolutely and it feels like control over AGI I say we live in a universe over AGI I say we live in a universe over AGI I say we live in a universe where that's possible we come up with where that's possible we come up with where that's possible we come up with ways to actually do that it's also ways to actually do that it's also ways to actually do that it's also scary because the collection of humans scary because the collection of humans scary because the collection of humans that have the control over AGI they that have the control over AGI they that have the control over AGI they become more powerful than the other become more powerful than the other become more powerful than the other humans and uh they can let that power humans and uh they can let that power humans and uh they can let that power get to their get to their get to their head and then a small selection of them head and then a small selection of them head and then a small selection of them back to back to back to Stalin uh start getting ideas and then Stalin uh start getting ideas and then Stalin uh start getting ideas and then eventually it's one person usually with eventually it's one person usually with eventually it's one person usually with a mustache or a funny hat that starts a mustache or a funny hat that starts a mustache or a funny hat that starts sort of making big speeches and then all sort of making big speeches and then all sort of making big speeches and then all of a sudden you live in a world that's of a sudden you live in a world that's of a sudden you live in a world that's either 1984 A Brave New either 1984 A Brave New either 1984 A Brave New World and uh always at war with somebody World and uh always at war with somebody World and uh always at war with somebody and you know this whole idea of control and you know this whole idea of control and you know this whole idea of control turned out to be uh actually also not turned out to be uh actually also not turned out to be uh actually also not beneficial to humanity so that's scary beneficial to humanity so that's scary beneficial to humanity so that's scary too it's actually worse because too it's actually worse because too it's actually worse because historically they all died this could be historically they all died this could be historically they all died this could be different this could be permanent different this could be permanent different this could be permanent dictatorship permanent suffering well dictatorship permanent suffering well dictatorship permanent suffering well the nice thing about humans it seems the nice thing about humans it seems the nice thing about humans it seems like it seems like the moment power like it seems like the moment power like it seems like the moment power starts corrupting their mind they can starts corrupting their mind they can starts corrupting their mind they can create a huge amount of suffering so create a huge amount of suffering so create a huge amount of suffering so there's Negative they can kill people there's Negative they can kill people there's Negative they can kill people make people suffer but then they become make people suffer but then they become make people suffer but then they become worse and worse at their worse and worse at their worse and worse at their job the it feels like the more evil you job the it feels like the more evil you job the it feels like the more evil you start start start doing like the at least they're doing like the at least they're doing like the at least they're incompetent yeah they well no they incompetent yeah they well no they incompetent yeah they well no they become more and more incompetent so they become more and more incompetent so they become more and more incompetent so they start start losing their grip on power

start start losing their grip on power start start losing their grip on power so like holding on to power is not a so like holding on to power is not a so like holding on to power is not a trivial thing it requires extreme trivial thing it requires extreme trivial thing it requires extreme competence which I I suppose and was competence which I I suppose and was competence which I I suppose and was good at it requires you to do evil and good at it requires you to do evil and good at it requires you to do evil and be competent at it or just get lucky and be competent at it or just get lucky and be competent at it or just get lucky and those systems help with that you have those systems help with that you have those systems help with that you have perfect surveillance you can do some perfect surveillance you can do some perfect surveillance you can do some mind reading I presume eventually it mind reading I presume eventually it mind reading I presume eventually it would be very hard would be very hard would be very hard to uh remove control from more capable to uh remove control from more capable to uh remove control from more capable systems over us and then it would be systems over us and then it would be systems over us and then it would be hard for humans to become the hackers hard for humans to become the hackers hard for humans to become the hackers that escape the control of the AGI that escape the control of the AGI that escape the control of the AGI because the AGI is so damn good and then because the AGI is so damn good and then because the AGI is so damn good and then yeah yeah yeah yeah yeah yeah yeah yeah yeah and then the the dictator is Immortal and then the the dictator is Immortal and then the the dictator is Immortal yeah this not great that's not a great yeah this not great that's not a great yeah this not great that's not a great outcome see I'm more afraid of humans outcome see I'm more afraid of humans outcome see I'm more afraid of humans than AI than AI than AI systems I'm afraid I believe that most systems I'm afraid I believe that most systems I'm afraid I believe that most humans want to do good and have the humans want to do good and have the humans want to do good and have the capacity to do good but also all humans capacity to do good but also all humans capacity to do good but also all humans have the capacity to do have the capacity to do have the capacity to do evil and um when you test them by giving evil and um when you test them by giving evil and um when you test them by giving them Absolute po as you would if you them Absolute po as you would if you them Absolute po as you would if you give them give them give them AGI that could result in a lot a lot of AGI that could result in a lot a lot of AGI that could result in a lot a lot of suffering what gives you hope about the future I what gives you hope about the future I could be wrong I've been wrong before if could be wrong I've been wrong before if could be wrong I've been wrong before if if you if you look 100 years from now if you if you look 100 years from now if you if you look 100 years from now and you're Immortal and you look back and you're Immortal and you look back and you're Immortal and you look back and it turns out this whole conversation and it turns out this whole conversation and it turns out this whole conversation you said a lot of things that were very you said a lot of things that were very you said a lot of things that were very wrong now that looking 100 100 years wrong now that looking 100 100 years wrong now that looking 100 100 years back what would be the explanation what back what would be the explanation what back what would be the explanation what happened in those 100 years that made

happened in those 100 years that made happened in those 100 years that made you wrong that made the words you said you wrong that made the words you said you wrong that made the words you said today wrong there is so many today wrong there is so many today wrong there is so many possibilities we had catastrophic events possibilities we had catastrophic events possibilities we had catastrophic events which prevented development of advanced which prevented development of advanced which prevented development of advanced microchips that's a hopeful future uh we microchips that's a hopeful future uh we microchips that's a hopeful future uh we could be in one of those personal could be in one of those personal could be in one of those personal universes and the one I'm in is universes and the one I'm in is universes and the one I'm in is beautiful it's all about me and I like beautiful it's all about me and I like beautiful it's all about me and I like it a lot so we've now just to linger on it a lot so we've now just to linger on it a lot so we've now just to linger on that that means like every every human that that means like every every human that that means like every every human has their personal Universe has their personal Universe has their personal Universe yes maybe multiple ones hey why not you yes maybe multiple ones hey why not you yes maybe multiple ones hey why not you shop around uh um it's possible that shop around uh um it's possible that shop around uh um it's possible that somebody comes up with somebody comes up with somebody comes up with alternative model for building AI which alternative model for building AI which alternative model for building AI which is not based on neural networks which is not based on neural networks which is not based on neural networks which are hard to scrutinize and that are hard to scrutinize and that are hard to scrutinize and that alternative is somehow I don't see how alternative is somehow I don't see how alternative is somehow I don't see how but somehow avoiding all the problems uh but somehow avoiding all the problems uh but somehow avoiding all the problems uh I speak about in general terms not I speak about in general terms not I speak about in general terms not applying them to specific applying them to specific applying them to specific architectures uh aliens come and give us architectures uh aliens come and give us architectures uh aliens come and give us friendly super intelligence there is so friendly super intelligence there is so friendly super intelligence there is so many options is it also possible that many options is it also possible that many options is it also possible that creating creating creating super intelligence systems becomes super intelligence systems becomes super intelligence systems becomes harder and harder so meaning harder and harder so meaning harder and harder so meaning like it's not so easy to do like it's not so easy to do like it's not so easy to do the uh fo the take the uh fo the take the uh fo the take off so that would probably speak more off so that would probably speak more off so that would probably speak more about how much smarter that system is about how much smarter that system is about how much smarter that system is compared to us so maybe it's hard to be compared to us so maybe it's hard to be compared to us so maybe it's hard to be a million times smarter but it's still a million times smarter but it's still a million times smarter but it's still okay to be five times smarter right so okay to be five times smarter right so okay to be five times smarter right so that is totally possible that I have no that is totally possible that I have no that is totally possible that I have no objections to so like it's there's a S

objections to so like it's there's a S objections to so like it's there's a S curve type situation about smarter and curve type situation about smarter and curve type situation about smarter and is going to be like 3.7 times smarter is going to be like 3.7 times smarter is going to be like 3.7 times smarter than all of human civilization right than all of human civilization right than all of human civilization right just the problems with face in this just the problems with face in this just the problems with face in this world each problem is like an IQ test world each problem is like an IQ test world each problem is like an IQ test you need certain intelligence to solve you need certain intelligence to solve you need certain intelligence to solve it so we just don't have more complex it so we just don't have more complex it so we just don't have more complex problems outside of mathematics for it problems outside of mathematics for it problems outside of mathematics for it to be showing off like you can have IQ to be showing off like you can have IQ to be showing off like you can have IQ of 500 if you're playing tic tac toe it of 500 if you're playing tic tac toe it of 500 if you're playing tic tac toe it doesn't show doesn't matter so the idea doesn't show doesn't matter so the idea doesn't show doesn't matter so the idea there is that the problems Define Your there is that the problems Define Your there is that the problems Define Your Capacity your cognitive capacity Capacity your cognitive capacity Capacity your cognitive capacity capacity so because the problems on capacity so because the problems on capacity so because the problems on Earth are not sufficiently difficult Earth are not sufficiently difficult Earth are not sufficiently difficult it's not going to be able to um expand it's not going to be able to um expand it's not going to be able to um expand its cognitive capacity possible and its cognitive capacity possible and its cognitive capacity possible and because of that wouldn't that be a good because of that wouldn't that be a good because of that wouldn't that be a good thing that it still could be a lot thing that it still could be a lot thing that it still could be a lot smarter than us and to dominate long smarter than us and to dominate long smarter than us and to dominate long term you just need some Advantage you term you just need some Advantage you term you just need some Advantage you have to be the smartest you don't have have to be the smartest you don't have have to be the smartest you don't have to be a million times smarter so even 5x to be a million times smarter so even 5x to be a million times smarter so even 5x might be enough it'd be impressive what might be enough it'd be impressive what might be enough it'd be impressive what is it IQ of a thousand I mean I know is it IQ of a thousand I mean I know is it IQ of a thousand I mean I know those units don't mean anything at that those units don't mean anything at that those units don't mean anything at that scale but still like as a comparison the scale but still like as a comparison the scale but still like as a comparison the smartest human is like 200 well actually smartest human is like 200 well actually smartest human is like 200 well actually no I didn't mean compared to an no I didn't mean compared to an no I didn't mean compared to an individual human I I meant compared to individual human I I meant compared to individual human I I meant compared to the collective intelligence the human the collective intelligence the human the collective intelligence the human species if you're somehow 5x smarter species if you're somehow 5x smarter species if you're somehow 5x smarter than than than that we are more productive as a group I that we are more productive as a group I that we are more productive as a group I don't think we are more capable of don't think we are more capable of don't think we are more capable of solving individual problems like if all solving individual problems like if all solving individual problems like if all of humanity plays chest together we are of humanity plays chest together we are of humanity plays chest together we are not like a million times better than not like a million times better than not like a million times better than world world world champion that's because the that there's

champion that's because the that there's champion that's because the that there's uh that's like One S curve is the CH uh that's like One S curve is the CH uh that's like One S curve is the CH chess but humanity is very good at chess but humanity is very good at chess but humanity is very good at exploring the full range of ideas like exploring the full range of ideas like exploring the full range of ideas like the more Einstein you have the more the the more Einstein you have the more the the more Einstein you have the more the just a high probability you come up with just a high probability you come up with just a high probability you come up with general relativity I feel like it's more general relativity I feel like it's more general relativity I feel like it's more about quantity super intelligence than about quantity super intelligence than about quantity super intelligence than quality super intelligence sure but you quality super intelligence sure but you quality super intelligence sure but you know quantity and and enough quantity know quantity and and enough quantity know quantity and and enough quantity sometimes becomes sometimes becomes sometimes becomes quality oh man humans uh what do you quality oh man humans uh what do you quality oh man humans uh what do you think is the meaning of uh this whole think is the meaning of uh this whole think is the meaning of uh this whole thing why we've been we've been talking thing why we've been we've been talking thing why we've been we've been talking about humans about humans about humans and not humans not dying but why are we and not humans not dying but why are we and not humans not dying but why are we here it's a simulation we're being here it's a simulation we're being here it's a simulation we're being tested the test is will you be dumb tested the test is will you be dumb tested the test is will you be dumb enough to create super intelligence and enough to create super intelligence and enough to create super intelligence and release it so the objective function is release it so the objective function is release it so the objective function is not be dumb enough to kill ourselves not be dumb enough to kill ourselves not be dumb enough to kill ourselves yeah you unsafe prove yourself to be a yeah you unsafe prove yourself to be a yeah you unsafe prove yourself to be a safe agent who doesn't do that and you safe agent who doesn't do that and you safe agent who doesn't do that and you get to go to the next game the next get to go to the next game the next get to go to the next game the next level of the game what's the next level level of the game what's the next level level of the game what's the next level I don't know I don't know I don't know I haven't hacked the simulation yet well I haven't hacked the simulation yet well I haven't hacked the simulation yet well maybe hacking the simulation is the maybe hacking the simulation is the maybe hacking the simulation is the thing I'm working as fast as I thing I'm working as fast as I thing I'm working as fast as I can and if physics would be a way to do can and if physics would be a way to do can and if physics would be a way to do that quantum physics yeah definitely that quantum physics yeah definitely that quantum physics yeah definitely well I hope we do and I hope whatever is well I hope we do and I hope whatever is well I hope we do and I hope whatever is outside is even more fun than this one outside is even more fun than this one outside is even more fun than this one cuz this one was pretty damn fun and uh cuz this one was pretty damn fun and uh cuz this one was pretty damn fun and uh just a big thank you for doing the work just a big thank you for doing the work just a big thank you for doing the work you're doing there's so much exciting you're doing there's so much exciting you're doing there's so much exciting development in Ai and to ground it in development in Ai and to ground it in development in Ai and to ground it in the um the existential risks is really the um the existential risks is really the um the existential risks is really really really really important humans love to create stuff

important humans love to create stuff important humans love to create stuff and we should be careful not to destroy and we should be careful not to destroy and we should be careful not to destroy ourselves in the process so thank you ourselves in the process so thank you ourselves in the process so thank you for doing that really important work for doing that really important work for doing that really important work thank you so much for inviting me it was thank you so much for inviting me it was thank you so much for inviting me it was amazing and my dream is to be proven amazing and my dream is to be proven amazing and my dream is to be proven wrong if everyone just you know picks up wrong if everyone just you know picks up wrong if everyone just you know picks up a paper or a book and shows how I messed a paper or a book and shows how I messed a paper or a book and shows how I messed it up that would be optimal but for now it up that would be optimal but for now it up that would be optimal but for now the simulation the simulation the simulation continues thank you Roman thanks for continues thank you Roman thanks for continues thank you Roman thanks for listening to this conversation with listening to this conversation with listening to this conversation with Roman yski to support this podcast Roman yski to support this podcast Roman yski to support this podcast please check out our sponsors in the please check out our sponsors in the please check out our sponsors in the description and now let me leave you description and now let me leave you description and now let me leave you with some words from Frank Herbert and with some words from Frank Herbert and with some words from Frank Herbert and dune I must not fear fear is the mind dune I must not fear fear is the mind dune I must not fear fear is the mind killer fear is The Little Death that killer fear is The Little Death that killer fear is The Little Death that brings total brings total brings total obliteration I will face fear I will obliteration I will face fear I will obliteration I will face fear I will permit it to pass over me and through me permit it to pass over me and through me permit it to pass over me and through me and when it has gone past I will turn and when it has gone past I will turn and when it has gone past I will turn the inner eye to see its path where the the inner eye to see its path where the the inner eye to see its path where the fear has gone fear has gone fear has gone there will be nothing only I will there will be nothing only I will there will be nothing only I will remain thank you for listening and hope remain thank you for listening and hope remain thank you for listening and hope to see you next time