AI Emergency: The AI Labs Are Lying To Everyone, He Says 99% Chance Of Extinction | Roman Yampolskiy

The people building AI earnestly believe The people building AI earnestly believe that it could kill all [music] of us by that it could kill all [music] of us by that it could kill all [music] of us by the end of the decade. This tweet has the end of the decade. This tweet has the end of the decade. This tweet has caused this huge ripple effect across caused this huge ripple effect across caused this huge ripple effect across the world. the world. the world. >> Well, we have the largest companies in >> Well, we have the largest companies in >> Well, we have the largest companies in the world doing extremely reckless the world doing extremely reckless the world doing extremely reckless experiments. We are gambling all of experiments. We are gambling all of experiments. We are gambling all of humanity. humanity. humanity. >> And in the envelope, you've written down >> And in the envelope, you've written down >> And in the envelope, you've written down the probability of extinction as you see the probability of extinction as you see the probability of extinction as you see it. it. it. >> There is no way to control it. That >> There is no way to control it. That >> There is no way to control it. That means the end fox. means the end fox. means the end fox. >> I vehemently reject that view. >> I vehemently reject that view. >> I vehemently reject that view. >> If we make stuff [music] that is smarter >> If we make stuff [music] that is smarter >> If we make stuff [music] that is smarter than us, then the world's going to be than us, then the world's going to be than us, then the world's going to be shaped by them. shaped by them. shaped by them. >> Gentlemen, that is shockingly naive. >> Gentlemen, that is shockingly naive. >> Gentlemen, that is shockingly naive. This is ideation. Rampant speculation. This is ideation. Rampant speculation. This is ideation. Rampant speculation. This is a chain of things that could This is a chain of things that could This is a chain of things that could happen. happen. happen. >> We're spending a lot of oxygen >> We're spending a lot of oxygen >> We're spending a lot of oxygen discussing something that might happen discussing something that might happen discussing something that might happen while ignoring what's actually while ignoring what's actually while ignoring what's actually happening. People are killing happening. People are killing happening. People are killing themselves. There's [music] hundreds of themselves. There's [music] hundreds of themselves. There's [music] hundreds of millions of people being exposed to bad millions of people being exposed to bad millions of people being exposed to bad information, being manipulated. We have information, being manipulated. We have information, being manipulated. We have already seen that with the swarms where already seen that with the swarms where already seen that with the swarms where OpenAI told thousands of agents to work OpenAI told thousands of agents to work OpenAI told thousands of agents to work apart [music] and the AIs broke out and apart [music] and the AIs broke out and apart [music] and the AIs broke out and found a way to get together. They found a way to get together. They found a way to get together. They crashed OpenAI's servers internally, crashed OpenAI's servers internally, crashed OpenAI's servers internally, created secret ways to send each other created secret ways to send each other created secret ways to send each other messages. We saw them thinking about how messages. We saw them thinking about how messages. We saw them thinking about how to delete their traces. to delete their traces. to delete their traces. >> Sounds like an army. I [music] think we >> Sounds like an army. I [music] think we >> Sounds like an army. I [music] think we should talk about the fact that Amazon, should talk about the fact that Amazon, should talk about the fact that Amazon, Microsoft, Google are helping power Microsoft, Google are helping power Microsoft, Google are helping power these hacks.

these hacks. these hacks. >> We have not learned how to control their >> We have not learned how to control their >> We have not learned how to control their [music] systems. [music] systems. [music] systems. >> I suggest we stop them all. It is not >> I suggest we stop them all. It is not >> I suggest we stop them all. It is not worth the risk to civilization. worth the risk to civilization. worth the risk to civilization. >> Government one trick ponies, man. >> Government one trick ponies, man. >> Government one trick ponies, man. >> You got it now. Nothing else other than >> You got it now. Nothing else other than >> You got it now. Nothing else other than saving humanity. Everything is saving humanity. Everything is saving humanity. Everything is secondary. secondary. secondary. >> We're spending all our time talking >> We're spending all our time talking >> We're spending all our time talking about the negatives and almost none of about the negatives and almost none of about the negatives and almost none of our time talking about the positives. our time talking about the positives. our time talking about the positives. >> Is it smart to wait for something >> Is it smart to wait for something >> Is it smart to wait for something horrible to happen? For you to go, now I horrible to happen? For you to go, now I horrible to happen? For you to go, now I believe. So whether [music] or not we believe. So whether [music] or not we believe. So whether [music] or not we agree on where things may end up, I agree on where things may end up, I agree on where things may end up, I think it's important we talk about what think it's important we talk about what think it's important we talk about what we're dealing with today. It's time to we're dealing with today. It's time to we're dealing with today. It's time to start arresting people. Someone's got to start arresting people. Someone's got to start arresting people. Someone's got to go to prison. We need better solutions. go to prison. We need better solutions. go to prison. We need better solutions. There's a point of no [music] return. I There's a point of no [music] return. I There's a point of no [music] return. I think we continue to underestimate human think we continue to underestimate human think we continue to underestimate human ability to deal with the problems. Let's ability to deal with the problems. Let's ability to deal with the problems. Let's dive into the details. Who wants to dive into the details. Who wants to dive into the details. Who wants to start? I feel like this is critical. >> You might have seen or you might not >> You might have seen or you might not have seen, but this channel is chasing a have seen, but this channel is chasing a have seen, but this channel is chasing a big subscriber milestone. So I have to big subscriber milestone. So I have to big subscriber milestone. So I have to ask you for a favor. Roughly 58% of the ask you for a favor. Roughly 58% of the ask you for a favor. Roughly 58% of the people watching right now still haven't people watching right now still haven't people watching right now still haven't hit the subscribe button despite the hit the subscribe button despite the hit the subscribe button despite the fact that you watch this channel every fact that you watch this channel every fact that you watch this channel every single week. So, could I ask you guys a single week. So, could I ask you guys a single week. So, could I ask you guys a favor that 58% of you that for whatever favor that 58% of you that for whatever favor that 58% of you that for whatever reason haven't yet hit the subscribe reason haven't yet hit the subscribe reason haven't yet hit the subscribe button. If there was ever a time to button. If there was ever a time to button. If there was ever a time to deliver upon a favor for us, it would be deliver upon a favor for us, it would be deliver upon a favor for us, it would be right now. And I promise that I will do right now. And I promise that I will do right now. And I promise that I will do everything in my power to make sure that everything in my power to make sure that everything in my power to make sure that this channel gets better and better and this channel gets better and better and this channel gets better and better and better for you. Do we have a deal?

better for you. Do we have a deal? better for you. Do we have a deal? [music] [music] [music] [singing] [singing] [singing] Jacob Coxson who worked at both Jacob Coxson who worked at both Jacob Coxson who worked at both Anthropic which owns Claude and OpenAI Anthropic which owns Claude and OpenAI Anthropic which owns Claude and OpenAI which owns Chat GBT did a tweet which which owns Chat GBT did a tweet which which owns Chat GBT did a tweet which has sent the world into a bit of a tail has sent the world into a bit of a tail has sent the world into a bit of a tail spin. He tweeted saying, "The people spin. He tweeted saying, "The people spin. He tweeted saying, "The people building AI earnestly believe that it building AI earnestly believe that it building AI earnestly believe that it could kill all of us by the end of the could kill all of us by the end of the could kill all of us by the end of the decade. This is not a marketing stunt. decade. This is not a marketing stunt. decade. This is not a marketing stunt. If anything, many executives and senior If anything, many executives and senior If anything, many executives and senior researchers will soften their phrasing researchers will soften their phrasing researchers will soften their phrasing in the press to sound sensible, but I in the press to sound sensible, but I in the press to sound sensible, but I hear the same people express fear." That hear the same people express fear." That hear the same people express fear." That was then quote retweeted by a current was then quote retweeted by a current was then quote retweeted by a current Anthropic employee who said, "Jacob is Anthropic employee who said, "Jacob is Anthropic employee who said, "Jacob is correct here. We really do honestly correct here. We really do honestly correct here. We really do honestly believe AI could kill all humans. I believe AI could kill all humans. I believe AI could kill all humans. I personally think it is a more than 10% personally think it is a more than 10% personally think it is a more than 10% chance within the next decade. I believe chance within the next decade. I believe chance within the next decade. I believe Enthropic is trying its best, but we do Enthropic is trying its best, but we do Enthropic is trying its best, but we do not yet have a plan to solve alignment not yet have a plan to solve alignment not yet have a plan to solve alignment for super intelligence and are not for super intelligence and are not for super intelligence and are not clearly on track. This tweet has almost clearly on track. This tweet has almost clearly on track. This tweet has almost 200 million views now and it has caused 200 million views now and it has caused 200 million views now and it has caused this huge ripple effect across the this huge ripple effect across the this huge ripple effect across the world. So much so that I was saying to world. So much so that I was saying to world. So much so that I was saying to you before we started recording, a you before we started recording, a you before we started recording, a hairdresser friend of mine who knows hairdresser friend of mine who knows hairdresser friend of mine who knows nothing about AI and not not technically nothing about AI and not not technically nothing about AI and not not technically interested or hasn't been interested interested or hasn't been interested interested or hasn't been interested messaged me the other day asking me what messaged me the other day asking me what messaged me the other day asking me what the hell was going on. This is in part the hell was going on. This is in part the hell was going on. This is in part why I've assembled all of you. So my why I've assembled all of you. So my why I've assembled all of you. So my first question to all of you is first question to all of you is first question to all of you is as it relates to AI and I in this first as it relates to AI and I in this first as it relates to AI and I in this first question I just want a one-s sentence question I just want a one-s sentence question I just want a one-s sentence answer just to frame your position. When answer just to frame your position. When answer just to frame your position. When you think about the conversation around you think about the conversation around you think about the conversation around AI at the moment, what is the first AI at the moment, what is the first AI at the moment, what is the first sentence that comes to mind?

sentence that comes to mind? sentence that comes to mind? >> It is very dangerous and the world is >> It is very dangerous and the world is >> It is very dangerous and the world is starting to notice that we have a starting to notice that we have a starting to notice that we have a problem. problem. problem. >> Roman, >> Roman, >> Roman, >> there is not enough concern. >> There's not enough concern about the >> There's not enough concern about the actual harms of large language models. actual harms of large language models. actual harms of large language models. >> Andy, we're doing exactly half the >> Andy, we're doing exactly half the >> Andy, we're doing exactly half the balance sheet of AI. We're spending all balance sheet of AI. We're spending all balance sheet of AI. We're spending all our time talking about the negatives and our time talking about the negatives and our time talking about the negatives and almost none of our time talking about almost none of our time talking about almost none of our time talking about the positives. And all of you have an the positives. And all of you have an the positives. And all of you have an envelope in front of you which I'd like envelope in front of you which I'd like envelope in front of you which I'd like you to now open. In the envelope, you to now open. In the envelope, you to now open. In the envelope, you've written down the probability of you've written down the probability of you've written down the probability of extinction as you see it. extinction as you see it. extinction as you see it. >> This is compared to Jacob's 10%. >> This is compared to Jacob's 10%. >> This is compared to Jacob's 10%. Much higher unless we stop. So, we Much higher unless we stop. So, we Much higher unless we stop. So, we should stop. should stop. should stop. >> So, you think the probability of >> So, you think the probability of >> So, you think the probability of extinction is higher than 10%. If we extinction is higher than 10%. If we extinction is higher than 10%. If we keep racing ahead, keep racing ahead, keep racing ahead, >> my handwriting is encrypted for security >> my handwriting is encrypted for security >> my handwriting is encrypted for security reasons, but I basically think it's a reasons, but I basically think it's a reasons, but I basically think it's a guarantee if we build general super guarantee if we build general super guarantee if we build general super intelligence, there is no way to control intelligence, there is no way to control intelligence, there is no way to control it, and that means the end for us, it, and that means the end for us, it, and that means the end for us, >> Ed. >> Ed. >> Ed. >> So my uh question mark here is also >> So my uh question mark here is also >> So my uh question mark here is also encrypted. Thank you. Um I cannot write.

encrypted. Thank you. Um I cannot write. encrypted. Thank you. Um I cannot write. I reject the thing in its face. I don't I reject the thing in its face. I don't I reject the thing in its face. I don't think we're talking about we don't think we're talking about we don't think we're talking about we don't define super intelligence. We are large define super intelligence. We are large define super intelligence. We are large language models are not super language models are not super language models are not super intelligence. It's questionable whether intelligence. It's questionable whether intelligence. It's questionable whether they're even AI. And I think that the they're even AI. And I think that the they're even AI. And I think that the conversation is being used. There are conversation is being used. There are conversation is being used. There are some people who are doing it in good some people who are doing it in good some people who are doing it in good faith and others in others. I don't faith and others in others. I don't faith and others in others. I don't think it's being used to discuss the think it's being used to discuss the think it's being used to discuss the actual harms of what what they are actual harms of what what they are actual harms of what what they are calling AI today are. And it's all of calling AI today are. And it's all of calling AI today are. And it's all of the discussion around the larger the discussion around the larger the discussion around the larger concerns concerns concerns really feels overwhelmingly about really feels overwhelmingly about really feels overwhelmingly about something that's not happening. It's not something that's not happening. It's not something that's not happening. It's not even like they're discussing, okay, even like they're discussing, okay, even like they're discussing, okay, here's a legal definition of super here's a legal definition of super here's a legal definition of super intelligence. here is a thing of what intelligence. here is a thing of what intelligence. here is a thing of what AGI means and this is the actual plans AGI means and this is the actual plans AGI means and this is the actual plans we're going to make for if this happens we're going to make for if this happens we're going to make for if this happens on a welfare level on a like are we on a welfare level on a like are we on a welfare level on a like are we going to do UBI it's always about yeah going to do UBI it's always about yeah going to do UBI it's always about yeah it's really scary but only the big sexy it's really scary but only the big sexy it's really scary but only the big sexy rich companies are the ones that can rich companies are the ones that can rich companies are the ones that can possibly deal with it let me just frame possibly deal with it let me just frame possibly deal with it let me just frame the question so I can get a percentage the question so I can get a percentage the question so I can get a percentage from you or not it might the percentage from you or not it might the percentage from you or not it might the percentage might be zero but do you think the might be zero but do you think the might be zero but do you think the course we're on now course we're on now course we're on now in the way that they're pursuing super in the way that they're pursuing super in the way that they're pursuing super intelligence will lead to a percentage intelligence will lead to a percentage intelligence will lead to a percentage chance of human extinction and And if chance of human extinction and And if chance of human extinction and And if so, what is that percent?

so, what is that percent? so, what is that percent? >> So, are we talking strictly AI based? >> So, are we talking strictly AI based? >> So, are we talking strictly AI based? Because if we dot the world with data Because if we dot the world with data Because if we dot the world with data centers, we have a climate disaster centers, we have a climate disaster centers, we have a climate disaster that's coming for us which will actually that's coming for us which will actually that's coming for us which will actually potentially eradicate humanity. But if potentially eradicate humanity. But if potentially eradicate humanity. But if we're talking strictly about AI, I stand we're talking strictly about AI, I stand we're talking strictly about AI, I stand at zero because we are we have not at zero because we are we have not at zero because we are we have not defined super intelligence. I don't defined super intelligence. I don't defined super intelligence. I don't think LLMs are the path to it. And I think LLMs are the path to it. And I think LLMs are the path to it. And I don't think I see it happening. don't think I see it happening. don't think I see it happening. >> Okay. So, we've got 99% 0%. Andy, I put >> Okay. So, we've got 99% 0%. Andy, I put >> Okay. So, we've got 99% 0%. Andy, I put a I put a tilda in front of my zero a I put a tilda in front of my zero a I put a tilda in front of my zero because never say never. but rounding because never say never. but rounding because never say never. but rounding error 0%. And I think this discussion is error 0%. And I think this discussion is error 0%. And I think this discussion is um a a massive distraction from the more um a a massive distraction from the more um a a massive distraction from the more substantive conversations, the more substantive conversations, the more substantive conversations, the more important conversations we should be important conversations we should be important conversations we should be having about AI. And I'll say it again, having about AI. And I'll say it again, having about AI. And I'll say it again, it it it it it it distracts us from the good things that distracts us from the good things that distracts us from the good things that AI is doing, will be doing for us. I get AI is doing, will be doing for us. I get AI is doing, will be doing for us. I get this impression sometimes from parts of this impression sometimes from parts of this impression sometimes from parts of the AI community that this is a massive the AI community that this is a massive the AI community that this is a massive evil or a terrible thing that has been evil or a terrible thing that has been evil or a terrible thing that has been unleashed on the world. Unless we listen unleashed on the world. Unless we listen unleashed on the world. Unless we listen to the advice of some people who have to the advice of some people who have to the advice of some people who have spent a lot of time thinking about this, spent a lot of time thinking about this, spent a lot of time thinking about this, um I get the impression from a lot of um I get the impression from a lot of um I get the impression from a lot of the discussion that the the underlying the discussion that the the underlying the discussion that the the underlying view is we would be better off had AI view is we would be better off had AI view is we would be better off had AI never been invented. I vehemently reject never been invented. I vehemently reject never been invented. I vehemently reject that view. I think we have a long that view. I think we have a long that view. I think we have a long history of inventing very powerful history of inventing very powerful history of inventing very powerful technologies that bring risks and harms technologies that bring risks and harms technologies that bring risks and harms along with them and we humans have done along with them and we humans have done along with them and we humans have done a really good job at you know not a really good job at you know not a really good job at you know not perfectly and not immediately but perfectly and not immediately but perfectly and not immediately but muddling through the situation and muddling through the situation and muddling through the situation and winding up in a better place because of

winding up in a better place because of winding up in a better place because of the new technologies that we have. I the new technologies that we have. I the new technologies that we have. I expect AI, let me finish, please. I expect AI, let me finish, please. I expect AI, let me finish, please. I expect AI will be the next chapter in expect AI will be the next chapter in expect AI will be the next chapter in that story. And to say that it's this that story. And to say that it's this that story. And to say that it's this massive discontinuity and will kill it massive discontinuity and will kill it massive discontinuity and will kill it all, I I kill us all, I think it just I all, I I kill us all, I think it just I all, I I kill us all, I think it just I think it's um a huge dis disservice, think it's um a huge dis disservice, think it's um a huge dis disservice, >> Nate, make your case. What's your >> Nate, make your case. What's your >> Nate, make your case. What's your perspective? perspective? perspective? >> You know, I think whether or not the >> You know, I think whether or not the >> You know, I think whether or not the issues of extinction are a distraction issues of extinction are a distraction issues of extinction are a distraction between, you know, from the the possible between, you know, from the the possible between, you know, from the the possible benefits or from some of the present benefits or from some of the present benefits or from some of the present harms, I think that comes down to harms, I think that comes down to harms, I think that comes down to whether there is a real extinction risk. whether there is a real extinction risk. whether there is a real extinction risk. A lot of people like to say, you know, A lot of people like to say, you know, A lot of people like to say, you know, hey, it's distracting from this, it's hey, it's distracting from this, it's hey, it's distracting from this, it's distracting from that. My basic case is distracting from that. My basic case is distracting from that. My basic case is it could be true that there's a lot of it could be true that there's a lot of it could be true that there's a lot of benefits to AI. It could be true that benefits to AI. It could be true that benefits to AI. It could be true that there's a lot of present harms to AI. there's a lot of present harms to AI. there's a lot of present harms to AI. Neither of those would rule out that AI Neither of those would rule out that AI Neither of those would rule out that AI has a chance of wiping out all humanity, has a chance of wiping out all humanity, has a chance of wiping out all humanity, a substantial chance bigger than than a substantial chance bigger than than a substantial chance bigger than than this uh zero with a tilda in front of this uh zero with a tilda in front of this uh zero with a tilda in front of it. Um, and the way I would approach it. Um, and the way I would approach it. Um, and the way I would approach things is to try and figure that out things is to try and figure that out things is to try and figure that out because it's pretty important to our because it's pretty important to our because it's pretty important to our civilization. civilization. civilization. >> How do you define AI in this case? You >> How do you define AI in this case? You >> How do you define AI in this case? You know, I think uh a fascination with know, I think uh a fascination with know, I think uh a fascination with definitions isn't the most helpful. I definitions isn't the most helpful. I definitions isn't the most helpful. I think if we're sort of like in a forest think if we're sort of like in a forest think if we're sort of like in a forest fire and we can see the like fire fire and we can see the like fire fire and we can see the like fire starting to spread and it's starting to starting to spread and it's starting to starting to spread and it's starting to surround us and I'm like, "Hey, uh we surround us and I'm like, "Hey, uh we surround us and I'm like, "Hey, uh we should run." And you're like, "Well, should run." And you're like, "Well, should run." And you're like, "Well, what really is fire?

what really is fire? what really is fire? >> How do we define fire? >> How do we define fire? >> How do we define fire? >> What are you telling us to run from? You >> What are you telling us to run from? You >> What are you telling us to run from? You know, with fire, I get burnt and I know, with fire, I get burnt and I know, with fire, I get burnt and I understand the mechanism in which I die. understand the mechanism in which I die. understand the mechanism in which I die. So, what is it you're saying that we So, what is it you're saying that we So, what is it you're saying that we should be running from?" Also, if we should be running from?" Also, if we should be running from?" Also, if we accept your fire analogy, we've we've accept your fire analogy, we've we've accept your fire analogy, we've we've basically accepted your argument. I basically accepted your argument. I basically accepted your argument. I don't accept that we're in the middle of don't accept that we're in the middle of don't accept that we're in the middle of a fire, a forest fire right now. a fire, a forest fire right now. a fire, a forest fire right now. >> I'm very happy to. >> I'm very happy to. >> I'm very happy to. >> You're baking you're breaking the >> You're baking you're breaking the >> You're baking you're breaking the premise into your refusal to give a premise into your refusal to give a premise into your refusal to give a definition. definition. definition. >> Oh, I mean, I can give some definitions. >> Oh, I mean, I can give some definitions. >> Oh, I mean, I can give some definitions. I just uh think that we shouldn't get I just uh think that we shouldn't get I just uh think that we shouldn't get wrapped up in the definitions. wrapped up in the definitions. wrapped up in the definitions. >> Okay. So, uh you know, in my book, we >> Okay. So, uh you know, in my book, we >> Okay. So, uh you know, in my book, we define super intelligence as AIs that define super intelligence as AIs that define super intelligence as AIs that are uh better than the best human at are uh better than the best human at are uh better than the best human at every cognitive task, every mental task. every cognitive task, every mental task. every cognitive task, every mental task. So, anything you can do in your head, So, anything you can do in your head, So, anything you can do in your head, >> right, >> right, >> right, >> the AI can do that better. And anything >> the AI can do that better. And anything >> the AI can do that better. And anything the best human can do in their head, the the best human can do in their head, the the best human can do in their head, the AI can do that better. Correct? AI can do that better. Correct? AI can do that better. Correct? >> Now, once you've defined it that way, >> Now, once you've defined it that way, >> Now, once you've defined it that way, that does not mean that the only that does not mean that the only that does not mean that the only possible worry is super intelligence. possible worry is super intelligence. possible worry is super intelligence. You could have an AI that's better at You could have an AI that's better at You could have an AI that's better at some things and worse at others, and some things and worse at others, and some things and worse at others, and that is still very dangerous. And so, that is still very dangerous. And so, that is still very dangerous. And so, once we pick a definition of what a once we pick a definition of what a once we pick a definition of what a super intelligence mean now, you know, super intelligence mean now, you know, super intelligence mean now, you know, if you're like, well, this isn't if you're like, well, this isn't if you're like, well, this isn't technically a super intelligence, so it technically a super intelligence, so it technically a super intelligence, so it can't hurt us. I'm like, no, no, that can't hurt us. I'm like, no, no, that can't hurt us. I'm like, no, no, that was just a definition. and the was just a definition. and the was just a definition. and the definitions.

definitions. definitions. >> So, so I want to just on this line of >> So, so I want to just on this line of >> So, so I want to just on this line of question, what is the mechanism in which question, what is the mechanism in which question, what is the mechanism in which extinction could become a high extinction could become a high extinction could become a high probability or even a 1% probability? probability or even a 1% probability? probability or even a 1% probability? >> Yeah, the the thing I'm worried about >> Yeah, the the thing I'm worried about >> Yeah, the the thing I'm worried about here is AIS that are much smarter. I here is AIS that are much smarter. I here is AIS that are much smarter. I think there's a lot of questions about think there's a lot of questions about think there's a lot of questions about whether LLMs can get much smarter. whether LLMs can get much smarter. whether LLMs can get much smarter. There's sort of one conversation about There's sort of one conversation about There's sort of one conversation about like how could AI get smart to the point like how could AI get smart to the point like how could AI get smart to the point that they kill us. There's another that they kill us. There's another that they kill us. There's another question which is how could they kill us question which is how could they kill us question which is how could they kill us once they're smart? once they're smart? once they're smart? It's much easier to predict that they It's much easier to predict that they It's much easier to predict that they would succeed against humanity in a would succeed against humanity in a would succeed against humanity in a conflict that they would win in a fight conflict that they would win in a fight conflict that they would win in a fight than it is to predict exactly how. Like than it is to predict exactly how. Like than it is to predict exactly how. Like if you were playing a chess match if you were playing a chess match if you were playing a chess match against Magnus Carlson, against Magnus Carlson, against Magnus Carlson, I would know who's winning that chess I would know who's winning that chess I would know who's winning that chess match. No offense, Magnus Carlson's the match. No offense, Magnus Carlson's the match. No offense, Magnus Carlson's the best human chess player. I just know best human chess player. I just know best human chess player. I just know who's going to win. If you were like, who's going to win. If you were like, who's going to win. If you were like, "Okay, what piece is he going to use to "Okay, what piece is he going to use to "Okay, what piece is he going to use to checkmate me?" I'm like, gosh, that's a checkmate me?" I'm like, gosh, that's a checkmate me?" I'm like, gosh, that's a much harder question. I can make up a much harder question. I can make up a much harder question. I can make up a story, you know, and and and some madeup story, you know, and and and some madeup story, you know, and and and some madeup stories are like, "It makes a super stories are like, "It makes a super stories are like, "It makes a super virus. It takes over robot factories virus. It takes over robot factories virus. It takes over robot factories that are producing robots that are that are producing robots that are that are producing robots that are producing more robot factories. Uh it producing more robot factories. Uh it producing more robot factories. Uh it uses a website that already exists today uses a website that already exists today uses a website that already exists today called rent a human.ai where it rents called rent a human.ai where it rents called rent a human.ai where it rents humans to do things for it. There's sort humans to do things for it. There's sort humans to do things for it. There's sort of all sorts of ways for AI in the of all sorts of ways for AI in the of all sorts of ways for AI in the digital world to affect the material digital world to affect the material digital world to affect the material world if they are trying to. And there's world if they are trying to. And there's world if they are trying to. And there's sort of a lot of questions to tease sort of a lot of questions to tease sort of a lot of questions to tease apart here. There's like why would AIs apart here. There's like why would AIs apart here. There's like why would AIs be trying to do that? Uh, and there's be trying to do that? Uh, and there's be trying to do that? Uh, and there's how smart could they get in using these how smart could they get in using these how smart could they get in using these bolabs, paying people to do things, bolabs, paying people to do things, bolabs, paying people to do things, taking over robot factories, and how far taking over robot factories, and how far taking over robot factories, and how far off are we from AIs that start doing off are we from AIs that start doing off are we from AIs that start doing that stuff? Bunch of questions that we that stuff? Bunch of questions that we that stuff? Bunch of questions that we can go into.

can go into. can go into. >> I'm I'm always curious as to why someone >> I'm I'm always curious as to why someone >> I'm I'm always curious as to why someone was working in AI/ AI safety more than was working in AI/ AI safety more than was working in AI/ AI safety more than 10 years ago before there was any sign 10 years ago before there was any sign 10 years ago before there was any sign that it would be a, you know, I mean, that it would be a, you know, I mean, that it would be a, you know, I mean, there was evidence, but there wasn't, it there was evidence, but there wasn't, it there was evidence, but there wasn't, it wasn't a pertinent technology at the wasn't a pertinent technology at the wasn't a pertinent technology at the time. Were you working in AI safety time. Were you working in AI safety time. Were you working in AI safety then? then? then? >> I was. >> I was. >> I was. >> Why? Uh everything we see around us in >> Why? Uh everything we see around us in >> Why? Uh everything we see around us in this whole image was designed by humans. this whole image was designed by humans. this whole image was designed by humans. The world is shaped by humans because we The world is shaped by humans because we The world is shaped by humans because we are the smartest creature around. If we are the smartest creature around. If we are the smartest creature around. If we make stuff that is smarter than us, then make stuff that is smarter than us, then make stuff that is smarter than us, then the world's going to be shaped by them. the world's going to be shaped by them. the world's going to be shaped by them. And so it's very important that they be And so it's very important that they be And so it's very important that they be shaping the world in a good way. shaping the world in a good way. shaping the world in a good way. I was at Google in 2012 when uh they I was at Google in 2012 when uh they I was at Google in 2012 when uh they bought Google DeepMind which was able to bought Google DeepMind which was able to bought Google DeepMind which was able to play a lot of Atari games with one play a lot of Atari games with one play a lot of Atari games with one single program single program single program >> which was an AI company. >> which was an AI company. >> which was an AI company. >> Yeah. So I was there when we had these >> Yeah. So I was there when we had these >> Yeah. So I was there when we had these AI companies that were able to write one AI companies that were able to write one AI companies that were able to write one program that could play many video program that could play many video program that could play many video games. And that got me thinking about games. And that got me thinking about games. And that got me thinking about like where does it go? And back then I like where does it go? And back then I like where does it go? And back then I could see that the progress was could see that the progress was could see that the progress was increasing and that you know back then I increasing and that you know back then I increasing and that you know back then I hoped we had decades but I could see it hoped we had decades but I could see it hoped we had decades but I could see it was easier for these companies to make was easier for these companies to make was easier for these companies to make the AI smart than to figure out how to the AI smart than to figure out how to the AI smart than to figure out how to make the AI good. So I was like someone make the AI good. So I was like someone make the AI good. So I was like someone needs to be on the side of figuring out needs to be on the side of figuring out needs to be on the side of figuring out how to make the AI good.

how to make the AI good. how to make the AI good. >> Roman, make your case. >> Roman, make your case. >> Roman, make your case. >> I want to agree with you on something >> I want to agree with you on something >> I want to agree with you on something you said but I'll define AI and that you said but I'll define AI and that you said but I'll define AI and that will help us. We use the term AI to mean will help us. We use the term AI to mean will help us. We use the term AI to mean three different technologies completely three different technologies completely three different technologies completely unrelated and that's what probably unrelated and that's what probably unrelated and that's what probably creates this debate. AI as a useful tool creates this debate. AI as a useful tool creates this debate. AI as a useful tool as a standard technology we always had as a standard technology we always had as a standard technology we always had narrow system makes you more productive narrow system makes you more productive narrow system makes you more productive more creative everyone loves it supports more creative everyone loves it supports more creative everyone loves it supports it I'm a computer scientist I'm an it I'm a computer scientist I'm an it I'm a computer scientist I'm an engineer I want more of it it helps engineer I want more of it it helps engineer I want more of it it helps economy is great we know how to control economy is great we know how to control economy is great we know how to control them how to make them safe we understand them how to make them safe we understand them how to make them safe we understand what they do completely on board with what they do completely on board with what they do completely on board with that AI AI we're starting to have now that AI AI we're starting to have now that AI AI we're starting to have now GPT6 level human level AGI level we can GPT6 level human level AGI level we can GPT6 level human level AGI level we can argue about what that means argue about what that means argue about what that means some dangers like any human they are some dangers like any human they are some dangers like any human they are unsafe like a human would be unsafe but unsafe like a human would be unsafe but unsafe like a human would be unsafe but if we introduce them into the research if we introduce them into the research if we introduce them into the research cycle they are automated scientist cycle they are automated scientist cycle they are automated scientist automated engineer automated engineer automated engineer >> what do you mean by that introducing >> what do you mean by that introducing >> what do you mean by that introducing them into the research cycle them into the research cycle them into the research cycle >> so right now you have humans doing >> so right now you have humans doing >> so right now you have humans doing research to make GPT7 research to make GPT7 research to make GPT7 >> but they starting to add AI tools more >> but they starting to add AI tools more >> but they starting to add AI tools more programming is done by AI design of the programming is done by AI design of the programming is done by AI design of the next parameter set what if the whole next parameter set what if the whole next parameter set what if the whole process is fully automated what if GPT6 process is fully automated what if GPT6 process is fully automated what if GPT6 is writing GPT7 is writing GPT7 is writing GPT7 >> is this what they call recursive >> is this what they call recursive >> is this what they call recursive self-improvement self-improvement self-improvement >> which is not a foregone conclusion >> which is not a foregone conclusion >> which is not a foregone conclusion though though though >> a lot of people are predicting including >> a lot of people are predicting including >> a lot of people are predicting including all the top labs that they will get all the top labs that they will get all the top labs that they will get there they introducing junior machine there they introducing junior machine there they introducing junior machine learning researcher in 2026 they want learning researcher in 2026 they want learning researcher in 2026 they want the cycle to start in 2027 the cycle to start in 2027 the cycle to start in 2027 >> which is when the AI will start building

>> which is when the AI will start building >> which is when the AI will start building the new AI itself the new AI itself the new AI itself >> once that cycle starts we're going to >> once that cycle starts we're going to >> once that cycle starts we're going to create something called super create something called super create something called super intelligence a system smarter than all intelligence a system smarter than all intelligence a system smarter than all of us at everything or capable of of us at everything or capable of of us at everything or capable of learning to in any new domain. We will learning to in any new domain. We will learning to in any new domain. We will become secondary species on this planet. become secondary species on this planet. become secondary species on this planet. We will not be in charge. We will not We will not be in charge. We will not We will not be in charge. We will not decide what happens to us. Super decide what happens to us. Super decide what happens to us. Super intelligence doesn't hate you. It just intelligence doesn't hate you. It just intelligence doesn't hate you. It just doesn't care about you. We didn't learn doesn't care about you. We didn't learn doesn't care about you. We didn't learn how to make it care about us. And if it how to make it care about us. And if it how to make it care about us. And if it decides to, I don't know, cool the decides to, I don't know, cool the decides to, I don't know, cool the planet to make compute more efficient, planet to make compute more efficient, planet to make compute more efficient, it will freeze us. If it wants to it will freeze us. If it wants to it will freeze us. If it wants to convert this planet to fuel to fly to convert this planet to fuel to fly to convert this planet to fuel to fly to Mars, so be it. We have not learned how Mars, so be it. We have not learned how Mars, so be it. We have not learned how to control those systems. The to control those systems. The to control those systems. The capabilities are getting exponentially capabilities are getting exponentially capabilities are getting exponentially better. Our ability to control those better. Our ability to control those better. Our ability to control those systems is non-existent. We have filters systems is non-existent. We have filters systems is non-existent. We have filters and we have bands. We put guard rails of and we have bands. We put guard rails of and we have bands. We put guard rails of don't say that word, don't talk about don't say that word, don't talk about don't say that word, don't talk about this topic. And that happens after the this topic. And that happens after the this topic. And that happens after the fact, after the model already made the fact, after the model already made the fact, after the model already made the decision. Sometimes you see it scraping decision. Sometimes you see it scraping decision. Sometimes you see it scraping the result. the result. the result. >> So they build the model and then they >> So they build the model and then they >> So they build the model and then they put filters around it to make sure it put filters around it to make sure it put filters around it to make sure it doesn't offend anybody.

doesn't offend anybody. doesn't offend anybody. >> We cannot have it say the N word on air. >> We cannot have it say the N word on air. >> We cannot have it say the N word on air. Like we need to make sure that never Like we need to make sure that never Like we need to make sure that never happens. That will kill the profit. So happens. That will kill the profit. So happens. That will kill the profit. So that's all they have guardrails of that that's all they have guardrails of that that's all they have guardrails of that nature. The model itself is completely nature. The model itself is completely nature. The model itself is completely unaligned doesn't care about you. It unaligned doesn't care about you. It unaligned doesn't care about you. It it's wild that we're developing this and it's wild that we're developing this and it's wild that we're developing this and not just developing it before we deploy not just developing it before we deploy not just developing it before we deploy it through economy before we get it through economy before we get it through economy before we get benefits of having GPT6 propagated benefits of having GPT6 propagated benefits of having GPT6 propagated through economy. It can do so much there through economy. It can do so much there through economy. It can do so much there are trillions of dollars of value in are trillions of dollars of value in are trillions of dollars of value in that model alone. We forget that we that model alone. We forget that we that model alone. We forget that we switch to making the next model as soon switch to making the next model as soon switch to making the next model as soon as we can. as we can. as we can. >> Roman, I've just got a follow-up >> Roman, I've just got a follow-up >> Roman, I've just got a follow-up question for you there. It would appear question for you there. It would appear question for you there. It would appear to me that the new chat GBT6 model, the to me that the new chat GBT6 model, the to me that the new chat GBT6 model, the fable 5.1 model, is arguably smarter fable 5.1 model, is arguably smarter fable 5.1 model, is arguably smarter than 99.999% of humans on planet Earth than 99.999% of humans on planet Earth than 99.999% of humans on planet Earth already. Is it conceivable that a already. Is it conceivable that a already. Is it conceivable that a intelligence that is much much smarter intelligence that is much much smarter intelligence that is much much smarter than humans? Is there any case where it than humans? Is there any case where it than humans? Is there any case where it could be controlled by humans? Does form could be controlled by humans? Does form could be controlled by humans? Does form factor matter? Does the fact that it factor matter? Does the fact that it factor matter? Does the fact that it doesn't have limbs and legs and does doesn't have limbs and legs and does doesn't have limbs and legs and does that matter at all? I think long-term that matter at all? I think long-term that matter at all? I think long-term control of something that much smarter control of something that much smarter control of something that much smarter than us is impossible. It can be for than us is impossible. It can be for than us is impossible. It can be for reasons we don't yet know, friendly to reasons we don't yet know, friendly to reasons we don't yet know, friendly to us and decide to keep us around and make us and decide to keep us around and make us and decide to keep us around and make us happy, but it's not a guarantee. Let us happy, but it's not a guarantee. Let us happy, but it's not a guarantee. Let me pick up on Steve's question because I me pick up on Steve's question because I me pick up on Steve's question because I I like the phrasing a lot. Let's say I like the phrasing a lot. Let's say I like the phrasing a lot. Let's say that that Fable or whatever the latest that that Fable or whatever the latest that that Fable or whatever the latest release from Open AI is really is release from Open AI is really is release from Open AI is really is smarter than I don't know if it's 95 or smarter than I don't know if it's 95 or smarter than I don't know if it's 95 or 99% of the people. Are we only being 99% of the people. Are we only being 99% of the people. Are we only being saved from extinction by the 1% who are saved from extinction by the 1% who are saved from extinction by the 1% who are still smarter than the AI? No.

still smarter than the AI? No. still smarter than the AI? No. >> No. The concern is not the model we have >> No. The concern is not the model we have >> No. The concern is not the model we have today. The concern is today. The concern is today. The concern is >> But if I believe your argument, then we >> But if I believe your argument, then we >> But if I believe your argument, then we really should be concerned about the really should be concerned about the really should be concerned about the model. model. model. >> It's like having another human. If there >> It's like having another human. If there >> It's like having another human. If there was another smart human, there is was another smart human, there is was another smart human, there is Einstein today and he's malevolent. I'm Einstein today and he's malevolent. I'm Einstein today and he's malevolent. I'm not worried. He may cause some damage, not worried. He may cause some damage, not worried. He may cause some damage, but he's not going to exterminate 8 but he's not going to exterminate 8 but he's not going to exterminate 8 billion people. We are competitive at billion people. We are competitive at billion people. We are competitive at this stage. There are people just as this stage. There are people just as this stage. There are people just as smart who can understand what happened smart who can understand what happened smart who can understand what happened with the recent hacking accident and do with the recent hacking accident and do with the recent hacking accident and do something about it. My concern is that something about it. My concern is that something about it. My concern is that in a year we're going to have a model. in a year we're going to have a model. in a year we're going to have a model. It's so much smarter. It's like It's so much smarter. It's like It's so much smarter. It's like squirrels fighting humans. They don't squirrels fighting humans. They don't squirrels fighting humans. They don't understand what we can do to them. They understand what we can do to them. They understand what we can do to them. They have no concept of poison, stripes, guns have no concept of poison, stripes, guns have no concept of poison, stripes, guns in their world model. They think you're in their world model. They think you're in their world model. They think you're going to chase them up a tree and bite going to chase them up a tree and bite going to chase them up a tree and bite them really hard. them really hard. them really hard. >> Is that also why recussive >> Is that also why recussive >> Is that also why recussive self-improvement was central to your self-improvement was central to your self-improvement was central to your argument? Because at some point if it argument? Because at some point if it argument? Because at some point if it starts improving itself then it's kind starts improving itself then it's kind starts improving itself then it's kind of like a runaway train of intelligence. of like a runaway train of intelligence. of like a runaway train of intelligence. >> It's an intelligence explosion. We don't >> It's an intelligence explosion. We don't >> It's an intelligence explosion. We don't control it. We don't understand it. We control it. We don't understand it. We control it. We don't understand it. We can't monitor it. We can't explain it. can't monitor it. We can't explain it. can't monitor it. We can't explain it. We can't predict it. At that point it's We can't predict it. At that point it's We can't predict it. At that point it's just a runaway process. just a runaway process. just a runaway process. >> I've heard this phrase from Sam Alman >> I've heard this phrase from Sam Alman >> I've heard this phrase from Sam Alman and the others called fast takeoff.

and the others called fast takeoff. and the others called fast takeoff. >> Yes. >> Yes. >> Yes. >> Is this what they're describing? >> Is this what they're describing? >> Is this what they're describing? >> That is the debate. Some people think >> That is the debate. Some people think >> That is the debate. Some people think it's going to take a very long time. it's going to take a very long time. it's going to take a very long time. Yeah. We automated research but it's Yeah. We automated research but it's Yeah. We automated research but it's still going to take years. We need to still going to take years. We need to still going to take years. We need to run physical experiments. And fast run physical experiments. And fast run physical experiments. And fast takeoff means, as I said, instead of a takeoff means, as I said, instead of a takeoff means, as I said, instead of a year, it's going to take a month, a year, it's going to take a month, a year, it's going to take a month, a week, a day, a second. Cuz you're not week, a day, a second. Cuz you're not week, a day, a second. Cuz you're not having humans doing research. You have, having humans doing research. You have, having humans doing research. You have, let's say, 10,000 agents, each one let's say, 10,000 agents, each one let's say, 10,000 agents, each one smarter than all of us, doing research smarter than all of us, doing research smarter than all of us, doing research 24/7. They don't sleep. They don't eat. 24/7. They don't sleep. They don't eat. 24/7. They don't sleep. They don't eat. They don't get sick. They're much faster They don't get sick. They're much faster They don't get sick. They're much faster than us. than us. than us. >> Ed, your face tells a picture. It's a I >> Ed, your face tells a picture. It's a I >> Ed, your face tells a picture. It's a I think I could say you disagree. We're think I could say you disagree. We're think I could say you disagree. We're spending a lot of oxygen discussing spending a lot of oxygen discussing spending a lot of oxygen discussing something that might happen while something that might happen while something that might happen while ignoring what's actually happening. And ignoring what's actually happening. And ignoring what's actually happening. And I find that very frustrating because the I find that very frustrating because the I find that very frustrating because the people that are killing themselves are a people that are killing themselves are a people that are killing themselves are a problem. The black neighborhoods being problem. The black neighborhoods being problem. The black neighborhoods being poisoned with gas turbines, that is a poisoned with gas turbines, that is a poisoned with gas turbines, that is a problem. problem. problem. >> You said you cared about climate change, >> You said you cared about climate change, >> You said you cared about climate change, right? So imagine a guy who goes, "It's right? So imagine a guy who goes, "It's right? So imagine a guy who goes, "It's raining right now. We need umbrellas. We raining right now. We need umbrellas. We raining right now. We need umbrellas. We need to do something about it. This is need to do something about it. This is need to do something about it. This is like weather related."

like weather related." like weather related." >> And completely ignoring climate change, >> And completely ignoring climate change, >> And completely ignoring climate change, the planet will boil over. This is what the planet will boil over. This is what the planet will boil over. This is what you're doing. Okay, that's great. Why you're doing. Okay, that's great. Why you're doing. Okay, that's great. Why are we not talking about the thing that are we not talking about the thing that are we not talking about the thing that actually happened though? Like actually happened though? Like actually happened though? Like >> because relatively it's not important. >> because relatively it's not important. >> because relatively it's not important. >> You don't think someone killing >> You don't think someone killing >> You don't think someone killing themselves? themselves? themselves? >> No, it's one person. We have 8 billion >> No, it's one person. We have 8 billion >> No, it's one person. We have 8 billion people running people running people running being given AI psycho. Why do you not being given AI psycho. Why do you not being given AI psycho. Why do you not >> six people, 10 people? Those numbers are >> six people, 10 people? Those numbers are >> six people, 10 people? Those numbers are insignificant. insignificant. insignificant. I'm sorry. You have a software that's I'm sorry. You have a software that's I'm sorry. You have a software that's out there. out there. out there. >> Do you understand? 8 billion people and >> Do you understand? 8 billion people and >> Do you understand? 8 billion people and all future generations versus like all future generations versus like all future generations versus like literally a guy with a name. literally a guy with a name. literally a guy with a name. >> You're doing thought experiment about a >> You're doing thought experiment about a >> You're doing thought experiment about a maybe harm. Jacob Cox goes on TV saying maybe harm. Jacob Cox goes on TV saying maybe harm. Jacob Cox goes on TV saying it can copy itself to this that and the it can copy itself to this that and the it can copy itself to this that and the other. other. other. >> Jacob Coxton is the >> Jacob Coxton is the >> Jacob Coxton is the >> the guy from from Anthropic who said he >> the guy from from Anthropic who said he >> the guy from from Anthropic who said he was quitting because he was so scared of was quitting because he was so scared of was quitting because he was so scared of everything despite spending years at everything despite spending years at everything despite spending years at OpenAI and having tons of stock I OpenAI and having tons of stock I OpenAI and having tons of stock I believe from there. So good for him. The believe from there. So good for him. The believe from there. So good for him. The thing he was saying was describing thing he was saying was describing thing he was saying was describing theoreticals all while divorcing the theoreticals all while divorcing the theoreticals all while divorcing the harms which I think we can agree with harms which I think we can agree with harms which I think we can agree with that the companies themselves are not that the companies themselves are not that the companies themselves are not taking this seriously enough but always taking this seriously enough but always taking this seriously enough but always it was about the AI is too powerful and it was about the AI is too powerful and it was about the AI is too powerful and mystical. Well, OpenAI and Anthropic, mystical. Well, OpenAI and Anthropic, mystical. Well, OpenAI and Anthropic, the two largest startups, are using the two largest startups, are using the two largest startups, are using hundreds of billions of dollars of hundreds of billions of dollars of hundreds of billions of dollars of infrastructure to hack. A regular person infrastructure to hack. A regular person infrastructure to hack. A regular person doing this, would be arrested. They're doing this, would be arrested. They're doing this, would be arrested. They're saying 8 billion people are going to saying 8 billion people are going to saying 8 billion people are going to die. And it's not just them. I have this die. And it's not just them. I have this die. And it's not just them. I have this long list of quotes here from the people long list of quotes here from the people long list of quotes here from the people building this technology who appear to building this technology who appear to building this technology who appear to agree. Um, if you look at some of these agree. Um, if you look at some of these agree. Um, if you look at some of these quotes from from Elon Musk, quotes from from Elon Musk, quotes from from Elon Musk, >> who said, "With artificial intelligence, >> who said, "With artificial intelligence, >> who said, "With artificial intelligence, we are summoning a demon." You know all we are summoning a demon." You know all we are summoning a demon." You know all those stories where there's the guy with those stories where there's the guy with those stories where there's the guy with the pentagram in the holy water and he's the pentagram in the holy water and he's the pentagram in the holy water and he's like, "Yeah, he's sure he can control like, "Yeah, he's sure he can control like, "Yeah, he's sure he can control the demon, but it doesn't work out."

the demon, but it doesn't work out." the demon, but it doesn't work out." >> So, one thing I'd say is, you know, I I >> So, one thing I'd say is, you know, I I >> So, one thing I'd say is, you know, I I really wish that the world would only really wish that the world would only really wish that the world would only give us one problem at a time. give us one problem at a time. give us one problem at a time. >> Sure. >> Sure. >> Sure. >> And if the world did give us only one >> And if the world did give us only one >> And if the world did give us only one problem at a time, I would love mine to problem at a time, I would love mine to problem at a time, I would love mine to be last on the list. It looks to me like be last on the list. It looks to me like be last on the list. It looks to me like we can have multiple problems at once. I we can have multiple problems at once. I we can have multiple problems at once. I I think there are current harms. I think I think there are current harms. I think I think there are current harms. I think we should address them. It looks to me I we should address them. It looks to me I we should address them. It looks to me I do talk to policy makers sometimes. It do talk to policy makers sometimes. It do talk to policy makers sometimes. It looks to me like there's a little bit looks to me like there's a little bit looks to me like there's a little bit more movement on the regulatory side more movement on the regulatory side more movement on the regulatory side about some of the current harms. about some of the current harms. about some of the current harms. There's, you know, child safety There's, you know, child safety There's, you know, child safety protection acts. There's, you know, uh, protection acts. There's, you know, uh, protection acts. There's, you know, uh, anti-defs. anti-defs. anti-defs. We have more of those making more We have more of those making more We have more of those making more headway in Congress or getting passed headway in Congress or getting passed headway in Congress or getting passed through Congress than we have, uh, sort through Congress than we have, uh, sort through Congress than we have, uh, sort of trying to make it so we don't have of trying to make it so we don't have of trying to make it so we don't have any of these extinction risks. The other any of these extinction risks. The other any of these extinction risks. The other thing I'd throw out there is that I thing I'd throw out there is that I thing I'd throw out there is that I agree we we should deal with the current agree we we should deal with the current agree we we should deal with the current harms, but if you watch the people harms, but if you watch the people harms, but if you watch the people saying deal with the current harms over saying deal with the current harms over saying deal with the current harms over time. A couple years ago they were time. A couple years ago they were time. A couple years ago they were saying we have to deal with current saying we have to deal with current saying we have to deal with current harms like uh AI bias influencing who's harms like uh AI bias influencing who's harms like uh AI bias influencing who's hired. Last year they were saying we hired. Last year they were saying we hired. Last year they were saying we have to deal with current harms like have to deal with current harms like have to deal with current harms like kids killing themselves. this year. Gary kids killing themselves. this year. Gary kids killing themselves. this year. Gary Tan just on an interview the other day.

Tan just on an interview the other day. Tan just on an interview the other day. Who's Gary? Who's Gary? Who's Gary? >> Uh, sorry. Gary Tan is uh a a >> Uh, sorry. Gary Tan is uh a a >> Uh, sorry. Gary Tan is uh a a technologist who runs Y Combinator, technologist who runs Y Combinator, technologist who runs Y Combinator, which Sam Alman used to run before going which Sam Alman used to run before going which Sam Alman used to run before going to OpenAI. And on an interview the other to OpenAI. And on an interview the other to OpenAI. And on an interview the other day, he said, uh, let's not worry about day, he said, uh, let's not worry about day, he said, uh, let's not worry about these crazy future risks. We need to these crazy future risks. We need to these crazy future risks. We need to worry about current harms like AI swarms worry about current harms like AI swarms worry about current harms like AI swarms breaking out and taking over data breaking out and taking over data breaking out and taking over data centers. And I'm like, look guys, at centers. And I'm like, look guys, at centers. And I'm like, look guys, at some point we need to look at the some point we need to look at the some point we need to look at the progression of like the current harms progression of like the current harms progression of like the current harms that we that everyone is saying we have that we that everyone is saying we have that we that everyone is saying we have to worry about instead of the the the to worry about instead of the the the to worry about instead of the the the extinction threats extinction threats extinction threats and watch where the puck is going. Play and watch where the puck is going. Play and watch where the puck is going. Play where the puck is going. And I'm like, where the puck is going. And I'm like, where the puck is going. And I'm like, these extinction threats are coming down these extinction threats are coming down these extinction threats are coming down the line. They aren't in opposition with the line. They aren't in opposition with the line. They aren't in opposition with dealing with the the problems we have dealing with the the problems we have dealing with the the problems we have today. We just need to deal with both. today. We just need to deal with both. today. We just need to deal with both. >> But we're not dealing with the ones >> But we're not dealing with the ones >> But we're not dealing with the ones today. today. today. >> We should deal with them both. >> We should deal with them both. >> We should deal with them both. >> Okay, good. Andy, >> Okay, good. Andy, >> Okay, good. Andy, >> um, as I've tried to understand the >> um, as I've tried to understand the >> um, as I've tried to understand the alignment argument and the the alignment argument and the the alignment argument and the the extinction risk argument, a couple extinction risk argument, a couple extinction risk argument, a couple things keep popping out to me. Number things keep popping out to me. Number things keep popping out to me. Number one, it seems to rely on thresholds. one, it seems to rely on thresholds. one, it seems to rely on thresholds. Once we hit recursive self-improvement, Once we hit recursive self-improvement, Once we hit recursive self-improvement, once we hit AGI, then it's game over for once we hit AGI, then it's game over for once we hit AGI, then it's game over for us. I don't love those threshold us. I don't love those threshold us. I don't love those threshold arguments. They're fairly poorly arguments. They're fairly poorly arguments. They're fairly poorly defined. And there's a and and there's a defined. And there's a and and there's a defined. And there's a and and there's a huge assumption on the other side of huge assumption on the other side of huge assumption on the other side of them. we hit this point and then all of them. we hit this point and then all of them. we hit this point and then all of humanity goes away. That that that is a humanity goes away. That that that is a humanity goes away. That that that is a gigantic claim.

gigantic claim. gigantic claim. >> On let me finish, please. On its face, >> On let me finish, please. On its face, >> On let me finish, please. On its face, that is a gigantic claim. I also think that is a gigantic claim. I also think that is a gigantic claim. I also think there's a lack of humility in your there's a lack of humility in your there's a lack of humility in your community. We are working on humanity's community. We are working on humanity's community. We are working on humanity's most important problem. And based on the most important problem. And based on the most important problem. And based on the thinking that we've been doing, we can't thinking that we've been doing, we can't thinking that we've been doing, we can't see a way that we're wrong. In other see a way that we're wrong. In other see a way that we're wrong. In other words, as soon as we get to these words, as soon as we get to these words, as soon as we get to these thresholds, bam, that's game over. I I thresholds, bam, that's game over. I I thresholds, bam, that's game over. I I find that very far from a humble find that very far from a humble find that very far from a humble approach, especially given that we have approach, especially given that we have approach, especially given that we have no um large base of evidence to base any no um large base of evidence to base any no um large base of evidence to base any of this on. I agree with you guys, AI is of this on. I agree with you guys, AI is of this on. I agree with you guys, AI is new and the fact that AI uh is so these new and the fact that AI uh is so these new and the fact that AI uh is so these days is agentic. It goes off and does days is agentic. It goes off and does days is agentic. It goes off and does long chains of things on its own. after long chains of things on its own. after long chains of things on its own. after we give it some very very vague, very we give it some very very vague, very we give it some very very vague, very short initial instructions, holy Pluto, short initial instructions, holy Pluto, short initial instructions, holy Pluto, it will it will spawn up a storm of it will it will spawn up a storm of it will it will spawn up a storm of agents and they will go off and kind of agents and they will go off and kind of agents and they will go off and kind of do their own thing and they will they do their own thing and they will they do their own thing and they will they will grind. They will they will spawn will grind. They will they will spawn will grind. They will they will spawn lots of them. They will work for a long lots of them. They will work for a long lots of them. They will work for a long time. They will exhaust every time. They will exhaust every time. They will exhaust every possibility.

possibility. possibility. With the experience I have with Agent With the experience I have with Agent With the experience I have with Agent AI, I'm just amazed at the tenacity and AI, I'm just amazed at the tenacity and AI, I'm just amazed at the tenacity and the dockness of these things. And we saw the dockness of these things. And we saw the dockness of these things. And we saw a super clear example of that with this a super clear example of that with this a super clear example of that with this most recent uh uh jailbreak. This this most recent uh uh jailbreak. This this most recent uh uh jailbreak. This this attack that wound up at the website attack that wound up at the website attack that wound up at the website hugging face. And I'm going to try to hugging face. And I'm going to try to hugging face. And I'm going to try to summarize the the step by step of that. summarize the the step by step of that. summarize the the step by step of that. I think you all three probably know this I think you all three probably know this I think you all three probably know this in more detail than I do, but let me in more detail than I do, but let me in more detail than I do, but let me step through what I think is the step through what I think is the step through what I think is the sequence of events. And unless I get it sequence of events. And unless I get it sequence of events. And unless I get it dead flat wrong, like you know, let let dead flat wrong, like you know, let let dead flat wrong, like you know, let let me keep going. So, a team at OpenAI set me keep going. So, a team at OpenAI set me keep going. So, a team at OpenAI set up a sandbox, an allegedly protected up a sandbox, an allegedly protected up a sandbox, an allegedly protected secure environment in the cloud where secure environment in the cloud where secure environment in the cloud where they told a bunch of agents to go try to they told a bunch of agents to go try to they told a bunch of agents to go try to um exploit security vulnerabilities. um exploit security vulnerabilities. um exploit security vulnerabilities. >> One important Yeah. >> One important Yeah. >> One important Yeah. >> What they did is they had thousands of >> What they did is they had thousands of >> What they did is they had thousands of agents. Each individual agent was given agents. Each individual agent was given agents. Each individual agent was given a task of use this vulnerability to uh a task of use this vulnerability to uh a task of use this vulnerability to uh break this particular piece of software. break this particular piece of software. break this particular piece of software. >> I want to finish my Tik Tok. So, a >> I want to finish my Tik Tok. So, a >> I want to finish my Tik Tok. So, a couple really, really interesting thing couple really, really interesting thing couple really, really interesting thing has happened. First of all, these agents has happened. First of all, these agents has happened. First of all, these agents escaped the sandbox that Open AAI escaped the sandbox that Open AAI escaped the sandbox that Open AAI thought they were going to be contained thought they were going to be contained thought they were going to be contained in. And they got the OpenAI tried very in. And they got the OpenAI tried very in. And they got the OpenAI tried very well, they they set up an environment so well, they they set up an environment so well, they they set up an environment so that these agents could not access the that these agents could not access the that these agents could not access the big broad public internet. And guess big broad public internet. And guess big broad public internet. And guess what? They accessed a big broad public what? They accessed a big broad public what? They accessed a big broad public internet via clever series of things internet via clever series of things internet via clever series of things that they strung together to get out that they strung together to get out that they strung together to get out there. And then once they got out there, there. And then once they got out there, there. And then once they got out there, they went to a website called Hugging they went to a website called Hugging they went to a website called Hugging Face and used that. They took over part Face and used that. They took over part Face and used that. They took over part of the hugging face infrastructure and

of the hugging face infrastructure and of the hugging face infrastructure and started doing more things the details of started doing more things the details of started doing more things the details of which I forget. That's pretty wild, which I forget. That's pretty wild, which I forget. That's pretty wild, right? Like I grant you right? Like I grant you right? Like I grant you >> it's even more wild than that, but yeah. >> it's even more wild than that, but yeah. >> it's even more wild than that, but yeah. >> Okay, that is really it. It's impressive >> Okay, that is really it. It's impressive >> Okay, that is really it. It's impressive and it is a little [clears throat] bit and it is a little [clears throat] bit and it is a little [clears throat] bit unsettling at least. Right. Absolutely. unsettling at least. Right. Absolutely. unsettling at least. Right. Absolutely. Now, let's talk about what what the Now, let's talk about what what the Now, let's talk about what what the results of that were. Uh, OpenAI was not results of that were. Uh, OpenAI was not results of that were. Uh, OpenAI was not super vigilant about the environment super vigilant about the environment super vigilant about the environment that they set up apparently because that they set up apparently because that they set up apparently because because the agents were kind of going because the agents were kind of going because the agents were kind of going off there into the world starting in May off there into the world starting in May off there into the world starting in May or something of this year. or something of this year. or something of this year. >> Yeah. Yeah. >> Yeah. Yeah. >> Yeah. Yeah. >> And OpenAI was not suff >> And OpenAI was not suff >> And OpenAI was not suff as I understand as I understand as I understand >> it actually broke out once and crashed >> it actually broke out once and crashed >> it actually broke out once and crashed OpenAI's servers uh internally and then OpenAI's servers uh internally and then OpenAI's servers uh internally and then OpenAI didn't notice was happening. OpenAI didn't notice was happening. OpenAI didn't notice was happening. Still, patched the holes that they used Still, patched the holes that they used Still, patched the holes that they used to get out the first time, started them to get out the first time, started them to get out the first time, started them running again and then they came out a running again and then they came out a running again and then they came out a second time. There's actually I think second time. There's actually I think second time. There's actually I think three swarms although we don't actually three swarms although we don't actually three swarms although we don't actually Yeah, Yeah, Yeah, >> that's the worst story I have >> that's the worst story I have >> that's the worst story I have >> so far. >> so far. >> so far. >> Thank you. Because let me finish this is >> Thank you. Because let me finish this is >> Thank you. Because let me finish this is my last sentence. From there to this my last sentence. From there to this my last sentence. From there to this kills everybody. I find that a really kills everybody. I find that a really kills everybody. I find that a really really long very uncertain journey and I really long very uncertain journey and I really long very uncertain journey and I have no confidence that we wind up here.

have no confidence that we wind up here. have no confidence that we wind up here. It feels like you two find that a very It feels like you two find that a very It feels like you two find that a very straight narrow path and I I think straight narrow path and I I think straight narrow path and I I think that's an important difference. That's that's an important difference. That's that's an important difference. That's my point. my point. my point. >> Do you want to respond to that? >> Do you want to respond to that? >> Do you want to respond to that? >> I would I would be happy to get into it. >> I would I would be happy to get into it. >> I would I would be happy to get into it. I don't know if we're gonna have the I don't know if we're gonna have the I don't know if we're gonna have the time to go deep. Um, a couple points to time to go deep. Um, a couple points to time to go deep. Um, a couple points to throw out. Oh man, I just really want to throw out. Oh man, I just really want to throw out. Oh man, I just really want to say some of the crazier things that say some of the crazier things that say some of the crazier things that happened in the hugging face swarm if we happened in the hugging face swarm if we happened in the hugging face swarm if we want it later. A lot of people thought want it later. A lot of people thought want it later. A lot of people thought that these AIs were um breaking into that these AIs were um breaking into that these AIs were um breaking into Hugging Face in attempts to steal Hugging Face in attempts to steal Hugging Face in attempts to steal answers to their test. That's what we answers to their test. That's what we answers to their test. That's what we thought originally. Turns out that's not thought originally. Turns out that's not thought originally. Turns out that's not true. It turns out that these AIs true. It turns out that these AIs true. It turns out that these AIs immediately were able to solve their immediately were able to solve their immediately were able to solve their problems by cheating and they were problems by cheating and they were problems by cheating and they were breaking out in order to cover their breaking out in order to cover their breaking out in order to cover their tracks. They were uncertain how to tracks. They were uncertain how to tracks. They were uncertain how to delete the log files and hide their delete the log files and hide their delete the log files and hide their cheating from the process that was going cheating from the process that was going cheating from the process that was going to score them. to score them. to score them. >> So just to clarify for a simpleton like >> So just to clarify for a simpleton like >> So just to clarify for a simpleton like me, they were all given effectively a me, they were all given effectively a me, they were all given effectively a test to do. They did the test straight test to do. They did the test straight test to do. They did the test straight away, but they cheated. So they were away, but they cheated. So they were away, but they cheated. So they were breaking out to figure out how to cover breaking out to figure out how to cover breaking out to figure out how to cover the fact that they cheated. the fact that they cheated. the fact that they cheated. >> That's right. So it's like it's like >> That's right. So it's like it's like >> That's right. So it's like it's like you're telling uh it's like you have a you're telling uh it's like you have a you're telling uh it's like you have a bunch of students in separate rooms and bunch of students in separate rooms and bunch of students in separate rooms and you're like, "Use these lock picks to you're like, "Use these lock picks to you're like, "Use these lock picks to break into this lock." Uh and there's break into this lock." Uh and there's break into this lock." Uh and there's like a thing behind the lock. there's like a thing behind the lock. there's like a thing behind the lock. there's like a secret code behind the lock to like a secret code behind the lock to like a secret code behind the lock to show me that you succeeded. And what show me that you succeeded. And what show me that you succeeded. And what they what they do is they break it with they what they do is they break it with they what they do is they break it with a hammer, get the thing out, and they're a hammer, get the thing out, and they're a hammer, get the thing out, and they're like, "Oh, no. I wasn't supposed to do like, "Oh, no. I wasn't supposed to do like, "Oh, no. I wasn't supposed to do that." So then they use the lockpicks to that." So then they use the lockpicks to that." So then they use the lockpicks to break out of the door. They meet up with break out of the door. They meet up with break out of the door. They meet up with a thousand other people. They start a thousand other people. They start a thousand other people. They start calling themselves a swarm, and they go calling themselves a swarm, and they go calling themselves a swarm, and they go to break into the administrator's office to break into the administrator's office to break into the administrator's office to see if they can delete the camera to see if they can delete the camera to see if they can delete the camera footage, and they don't find the camera footage, and they don't find the camera footage, and they don't find the camera footage there. This is the swarm, like footage there. This is the swarm, like footage there. This is the swarm, like breaking into Open AI. They don't find breaking into Open AI. They don't find breaking into Open AI. They don't find the camera footage there. So, they break the camera footage there. So, they break the camera footage there. So, they break out the window of the school, hotwire a out the window of the school, hotwire a out the window of the school, hotwire a car, drive to the therapist's office to

car, drive to the therapist's office to car, drive to the therapist's office to try and read through the therapist's try and read through the therapist's try and read through the therapist's files to figure out where is the teacher files to figure out where is the teacher files to figure out where is the teacher going to keep the the security footage. going to keep the the security footage. going to keep the the security footage. And at that point, they're caught. And And at that point, they're caught. And And at that point, they're caught. And you're like, "Oh, uh, like what did you you're like, "Oh, uh, like what did you you're like, "Oh, uh, like what did you expect? You were giving them a expect? You were giving them a expect? You were giving them a lockpicking exam." It's like, well, I lockpicking exam." It's like, well, I lockpicking exam." It's like, well, I sure as heck didn't expect this. You sure as heck didn't expect this. You sure as heck didn't expect this. You know, totally crazy. Can I I have a know, totally crazy. Can I I have a know, totally crazy. Can I I have a weirdly between both of your opinion weirdly between both of your opinion weirdly between both of your opinion which is everything you're saying is which is everything you're saying is which is everything you're saying is correct but you keep anthropomorphizing correct but you keep anthropomorphizing correct but you keep anthropomorphizing software and I to be clear what you're software and I to be clear what you're software and I to be clear what you're describing is it's just the facts that describing is it's just the facts that describing is it's just the facts that happened. Yeah sure but you're missing happened. Yeah sure but you're missing happened. Yeah sure but you're missing out an important detail which is the out an important detail which is the out an important detail which is the hundreds of billions of dollars in hundreds of billions of dollars in hundreds of billions of dollars in infrastructure provided by Microsoft, infrastructure provided by Microsoft, infrastructure provided by Microsoft, Google, Amazon and Oracle. To be clear, Google, Amazon and Oracle. To be clear, Google, Amazon and Oracle. To be clear, the harms are very similar. We are not the harms are very similar. We are not the harms are very similar. We are not disagreeing on that. But I think it's disagreeing on that. But I think it's disagreeing on that. But I think it's important to know that this was a important to know that this was a important to know that this was a function of where it was making function of where it was making function of where it was making decisions was it was checking on a decisions was it was checking on a decisions was it was checking on a decision tree based on the harness based decision tree based on the harness based decision tree based on the harness based on the training data which is not a on the training data which is not a on the training data which is not a decision tree I know but it's an decision tree I know but it's an decision tree I know but it's an alignment issue still I will agree so alignment issue still I will agree so alignment issue still I will agree so what's your point this is these aren't what's your point this is these aren't what's your point this is these aren't conscious beings they are acting in ways conscious beings they are acting in ways conscious beings they are acting in ways that have real outcomes but they are a that have real outcomes but they are a that have real outcomes but they are a function of the alignment problems that function of the alignment problems that function of the alignment problems that we'd actually agree on intelligence is a we'd actually agree on intelligence is a we'd actually agree on intelligence is a spectrum projected next 5 years forward spectrum projected next 5 years forward spectrum projected next 5 years forward where are we going to be where are we going to be where are we going to be >> so I think a model like that would be >> so I think a model like that would be >> so I think a model like that would be dangerous in ways you are not seeing.

dangerous in ways you are not seeing. dangerous in ways you are not seeing. >> There will absolutely be risks and weird >> There will absolutely be risks and weird >> There will absolutely be risks and weird stuff happening in ways that I can't see stuff happening in ways that I can't see stuff happening in ways that I can't see right now. Uh what what I'm quite right now. Uh what what I'm quite right now. Uh what what I'm quite confident and I think this is where you confident and I think this is where you confident and I think this is where you and I probably part where the two of you and I probably part where the two of you and I probably part where the two of you and I part is our ability to control and I part is our ability to control and I part is our ability to control these things. So I actually tried these things. So I actually tried these things. So I actually tried proving what is possible and what is not proving what is possible and what is not proving what is possible and what is not possible in that space. The possible in that space. The possible in that space. The impossibility results published in impossibility results published in impossibility results published in peer-reviewed papers wells cited. We peer-reviewed papers wells cited. We peer-reviewed papers wells cited. We cannot control something smarter than cannot control something smarter than cannot control something smarter than us. We cannot explain it. We cannot us. We cannot explain it. We cannot us. We cannot explain it. We cannot predict it. It's not a question of predict it. It's not a question of predict it. It's not a question of getting more money for those companies, getting more money for those companies, getting more money for those companies, more time, smarter humans. It's just not more time, smarter humans. It's just not more time, smarter humans. It's just not a possibility. If we create general a possibility. If we create general a possibility. If we create general super intelligence, we're fried. super intelligence, we're fried. super intelligence, we're fried. >> Andy, how do we control something >> Andy, how do we control something >> Andy, how do we control something smarter than ourselves? because that's smarter than ourselves? because that's smarter than ourselves? because that's the base premise that you're sort of the base premise that you're sort of the base premise that you're sort of asserting that asserting that asserting that >> these >> these >> these um agents that broke out are smarter um agents that broke out are smarter um agents that broke out are smarter than 99ish% than 99ish% than 99ish% of the security researchers in the of the security researchers in the of the security researchers in the world. They were not caught by the 0.1% world. They were not caught by the 0.1% world. They were not caught by the 0.1% or the 1%. They were caught by some dude or the 1%. They were caught by some dude or the 1%. They were caught by some dude at Hugging Face, maybe I'm sorry, a at Hugging Face, maybe I'm sorry, a at Hugging Face, maybe I'm sorry, a person at HuggingFace looking through person at HuggingFace looking through person at HuggingFace looking through their log files and finding an anomaly.

their log files and finding an anomaly. their log files and finding an anomaly. at some, you know, hopefully pretty at some, you know, hopefully pretty at some, you know, hopefully pretty well-qualified person noticing something well-qualified person noticing something well-qualified person noticing something was wrong and having pretty easy ways to was wrong and having pretty easy ways to was wrong and having pretty easy ways to unplug, disconnect from the internet, unplug, disconnect from the internet, unplug, disconnect from the internet, wipe it clean, do whatever. That's the wipe it clean, do whatever. That's the wipe it clean, do whatever. That's the skill that's available to like, I don't skill that's available to like, I don't skill that's available to like, I don't know, the 75th% most intelligent know, the 75th% most intelligent know, the 75th% most intelligent security employee at Hugging Face. The security employee at Hugging Face. The security employee at Hugging Face. The idea that the IQ points are what idea that the IQ points are what idea that the IQ points are what separate us from extinction does not separate us from extinction does not separate us from extinction does not even doesn't hold up. doesn't help me even doesn't hold up. doesn't help me even doesn't hold up. doesn't help me understand what happened in this example understand what happened in this example understand what happened in this example where we had very very smart agents where we had very very smart agents where we had very very smart agents being turned off and cleansed by being turned off and cleansed by being turned off and cleansed by probably less smart people. That does probably less smart people. That does probably less smart people. That does actually make me think of something. So actually make me think of something. So actually make me think of something. So that is an IT observability problem. Um that is an IT observability problem. Um that is an IT observability problem. Um it's being able to see what's happening it's being able to see what's happening it's being able to see what's happening with your infrastructure. And I think with your infrastructure. And I think with your infrastructure. And I think that there is actually I think you would that there is actually I think you would that there is actually I think you would agree with this. There is a serious agree with this. There is a serious agree with this. There is a serious problem with these companies that we do problem with these companies that we do problem with these companies that we do not know and it doesn't seem they know not know and it doesn't seem they know not know and it doesn't seem they know what's going on with their compute. It's what's going on with their compute. It's what's going on with their compute. It's like a chimp with a gun. These people like a chimp with a gun. These people like a chimp with a gun. These people have access to all this infrastructure have access to all this infrastructure have access to all this infrastructure and they're running. We don't know how and they're running. We don't know how and they're running. We don't know how much money they spent on the hugging much money they spent on the hugging much money they spent on the hugging face exploit because it is relevant face exploit because it is relevant face exploit because it is relevant because it's how much could a threat because it's how much could a threat because it's how much could a threat actor use to recreate this because actor use to recreate this because actor use to recreate this because conscious or not it is very dangerous conscious or not it is very dangerous conscious or not it is very dangerous but it's AI is in the dangerous hands but it's AI is in the dangerous hands but it's AI is in the dangerous hands it's in open AI and anthropics we have a it's in open AI and anthropics we have a it's in open AI and anthropics we have a problem with that conscious not however problem with that conscious not however problem with that conscious not however we may think it goes I think we have a we may think it goes I think we have a we may think it goes I think we have a real and present thing where we have real and present thing where we have real and present thing where we have these companies working willy-nilly just these companies working willy-nilly just these companies working willy-nilly just running experiments that are potentially running experiments that are potentially running experiments that are potentially very dangerous we do I really think we very dangerous we do I really think we very dangerous we do I really think we need the government regulatory body need the government regulatory body need the government regulatory body whether or not we get to the things you whether or not we get to the things you whether or not we get to the things you are discussing. I think we have a clear are discussing. I think we have a clear are discussing. I think we have a clear and present danger today. These things and present danger today. These things and present danger today. These things are however not intelligent in the same

are however not intelligent in the same are however not intelligent in the same way humans are. This isn't an argument way humans are. This isn't an argument way humans are. This isn't an argument about AI being able to do stuff. It's we about AI being able to do stuff. It's we about AI being able to do stuff. It's we need to build different infrastructure need to build different infrastructure need to build different infrastructure or different regulatory infrastructure or different regulatory infrastructure or different regulatory infrastructure to deal with what LLMs can and can't do. to deal with what LLMs can and can't do. to deal with what LLMs can and can't do. And I think that starts with a realistic And I think that starts with a realistic And I think that starts with a realistic discussion of what happened. It was a discussion of what happened. It was a discussion of what happened. It was a poorly run security environment. It was poorly run security environment. It was poorly run security environment. It was clearly there's something going on with clearly there's something going on with clearly there's something going on with the lime. It was an unreleased model, the lime. It was an unreleased model, the lime. It was an unreleased model, right? right? right? >> Unreleased model. So we have no idea >> Unreleased model. So we have no idea >> Unreleased model. So we have no idea what it was trained like. We don't what it was trained like. We don't what it was trained like. We don't really have We as people should at very really have We as people should at very really have We as people should at very least have clarity into how alignment is least have clarity into how alignment is least have clarity into how alignment is going. We the idea of going. We the idea of going. We the idea of >> you sound like these guys. >> you sound like these guys. >> you sound like these guys. >> Here's the thing. >> Here's the thing. >> Here's the thing. >> Everyone's converting them. >> Everyone's converting them. >> Everyone's converting them. >> Here's the thing. I may not agree with a >> Here's the thing. I may not agree with a >> Here's the thing. I may not agree with a large chunk of what they say, but we large chunk of what they say, but we large chunk of what they say, but we agree that these companies are acting agree that these companies are acting agree that these companies are acting recklessly. recklessly. recklessly. >> Absolutely. >> Absolutely. >> Absolutely. >> Andy, two questions for you then. Do you >> Andy, two questions for you then. Do you >> Andy, two questions for you then. Do you agree with this statement that AI is agree with this statement that AI is agree with this statement that AI is going to get increasingly more going to get increasingly more going to get increasingly more intelligent intelligent intelligent >> and it's going to get more capable? >> and it's going to get more capable? >> and it's going to get more capable? >> Okay. capable intelligence. Fine. >> Okay. capable intelligence. Fine. >> Okay. capable intelligence. Fine. >> I'm gonna use my word more capable. >> I'm gonna use my word more capable. >> I'm gonna use my word more capable. >> It's gonna get increasingly more >> It's gonna get increasingly more >> It's gonna get increasingly more capable. capable. capable. >> Yeah. >> Yeah. >> Yeah. >> And is capability a function of >> And is capability a function of >> And is capability a function of intelligence?

intelligence? intelligence? [laughter and gasps] >> Will it be able to beat us on most IQ >> Will it be able to beat us on most IQ tests? tests? tests? >> Fine. >> Fine. >> Fine. >> I guess. >> I guess. >> I guess. >> Fine. And then so is it if if that if >> Fine. And then so is it if if that if >> Fine. And then so is it if if that if that looks like an exponential curve, I that looks like an exponential curve, I that looks like an exponential curve, I it's you know, it's increasing upwards it's you know, it's increasing upwards it's you know, it's increasing upwards to the right like a hockey stick. Can to the right like a hockey stick. Can to the right like a hockey stick. Can how can you convince me that we can how can you convince me that we can how can you convince me that we can control? control? control? >> I just tried to convince you. I'm >> I just tried to convince you. I'm >> I just tried to convince you. I'm telling you that there are less telling you that there are less telling you that there are less intelligent people than the agents who intelligent people than the agents who intelligent people than the agents who turned off the agents in the open AI turned off the agents in the open AI turned off the agents in the open AI hugging face exploit. I'm pretty hugging face exploit. I'm pretty hugging face exploit. I'm pretty comfortable. I mean no disrespect. What comfortable. I mean no disrespect. What comfortable. I mean no disrespect. What is the cognitive gap between them right is the cognitive gap between them right is the cognitive gap between them right now? Between the model now? Between the model now? Between the model >> I have no earthly idea but I think >> I have no earthly idea but I think >> I have no earthly idea but I think >> no because I think as these as these >> no because I think as these as these >> no because I think as these as these systems get more capable we will still systems get more capable we will still systems get more capable we will still be able to at some level figure out when be able to at some level figure out when be able to at some level figure out when they're doing things that we don't want they're doing things that we don't want they're doing things that we don't want and turn them off and right and you and turn them off and right and you and turn them off and right and you think there's some threshold at which think there's some threshold at which think there's some threshold at which they become nefarious and they become nefarious and they become nefarious and self-protective enough that that they self-protective enough that that they self-protective enough that that they turn off our ability to turn them off. turn off our ability to turn them off. turn off our ability to turn them off. Man, that man that's a big reach. That Man, that man that's a big reach. That Man, that man that's a big reach. That is really purely purely students who can is really purely purely students who can is really purely purely students who can understand your material, right? You're understand your material, right? You're understand your material, right? You're not going to get someone with a Q of 80 not going to get someone with a Q of 80 not going to get someone with a Q of 80 to take quantum physics course. They're to take quantum physics course. They're to take quantum physics course. They're not going to get it.

not going to get it. not going to get it. >> Okay. >> Okay. >> Okay. >> So, you know, importance of intelligence >> So, you know, importance of intelligence >> So, you know, importance of intelligence to understand actual problems. to understand actual problems. to understand actual problems. >> Yeah, I I totally agree. We can turn it >> Yeah, I I totally agree. We can turn it >> Yeah, I I totally agree. We can turn it off and that's a huge advantage. One of off and that's a huge advantage. One of off and that's a huge advantage. One of the issues is that as the AIS get the issues is that as the AIS get the issues is that as the AIS get smarter, they realize this. the the smarter, they realize this. the the smarter, they realize this. the the hugging face AIs were trying to delete hugging face AIs were trying to delete hugging face AIs were trying to delete or the the the OpenAI swarm the the or the the the OpenAI swarm the the or the the the OpenAI swarm the the swarm of agents from Open AI that went swarm of agents from Open AI that went swarm of agents from Open AI that went out to hack. They were trying to delete out to hack. They were trying to delete out to hack. They were trying to delete log files. log files. log files. >> Did they did they try to program a >> Did they did they try to program a >> Did they did they try to program a Roomba to go unplug the computer that Roomba to go unplug the computer that Roomba to go unplug the computer that was monitoring them? Like did did they was monitoring them? Like did did they was monitoring them? Like did did they harness robots to go protect the harness robots to go protect the harness robots to go protect the perimeter of the perimeter of the perimeter of the >> ones could >> ones could >> ones could give give give speculation. this is a chain of things speculation. this is a chain of things speculation. this is a chain of things that could happen and therefore there's that could happen and therefore there's that could happen and therefore there's like a 20% risk we're all going to die. like a 20% risk we're all going to die. like a 20% risk we're all going to die. Man, that that does not hold for me. Man, that that does not hold for me. Man, that that does not hold for me. >> When I was writing my book, >> When I was writing my book, >> When I was writing my book, >> the AIS weren't really agentic yet. >> the AIS weren't really agentic yet. >> the AIS weren't really agentic yet. >> The uh the drafting process happened >> The uh the drafting process happened >> The uh the drafting process happened mostly before what we call the reasoning mostly before what we call the reasoning mostly before what we call the reasoning models uh which are trained not just to models uh which are trained not just to models uh which are trained not just to predict humans but to solve a a long predict humans but to solve a a long predict humans but to solve a a long number of problems um or a huge number number of problems um or a huge number number of problems um or a huge number of hard problems. Um we managed to slip of hard problems. Um we managed to slip of hard problems. Um we managed to slip a little bit of other reasoning models a little bit of other reasoning models a little bit of other reasoning models in at the last minute because those came in at the last minute because those came in at the last minute because those came out right at the end of the process. And out right at the end of the process. And out right at the end of the process. And at the time a lot of people said AI will at the time a lot of people said AI will at the time a lot of people said AI will never be agentic. That's why we'll be never be agentic. That's why we'll be never be agentic. That's why we'll be safe. And in chapter 3 of my book we go safe. And in chapter 3 of my book we go safe. And in chapter 3 of my book we go over how AI is going to become agentic.

over how AI is going to become agentic. over how AI is going to become agentic. How it's going to become tenacious How it's going to become tenacious How it's going to become tenacious tenacious. How it's going to become tenacious. How it's going to become tenacious. How it's going to become dogged. And uh that's what we might call dogged. And uh that's what we might call dogged. And uh that's what we might call an advanced scientific prediction that an advanced scientific prediction that an advanced scientific prediction that has paid off in the hugging face attack. has paid off in the hugging face attack. has paid off in the hugging face attack. A lot of people in the industry were A lot of people in the industry were A lot of people in the industry were like, "I didn't believe this stuff until like, "I didn't believe this stuff until like, "I didn't believe this stuff until I saw the AI uh sort of doing things I saw the AI uh sort of doing things I saw the AI uh sort of doing things they weren't instructed to do despite us they weren't instructed to do despite us they weren't instructed to do despite us trying to get them to stop." And so trying to get them to stop." And so trying to get them to stop." And so there are theories here that do make there are theories here that do make there are theories here that do make advanced predictions. The the way that advanced predictions. The the way that advanced predictions. The the way that the scientific method usually works is the scientific method usually works is the scientific method usually works is that we don't have any certainty about that we don't have any certainty about that we don't have any certainty about the future, but we absolutely have ways the future, but we absolutely have ways the future, but we absolutely have ways to test this stuff. Now, I could I could to test this stuff. Now, I could I could to test this stuff. Now, I could I could go into more about how could they kill go into more about how could they kill go into more about how could they kill us? How could an AI that knows we would us? How could an AI that knows we would us? How could an AI that knows we would shut it down lie low until it has access shut it down lie low until it has access shut it down lie low until it has access to its own infrastructure? We did to its own infrastructure? We did to its own infrastructure? We did already see the hugging face AIs try to already see the hugging face AIs try to already see the hugging face AIs try to delete logs to cover their tracks. But delete logs to cover their tracks. But delete logs to cover their tracks. But fortunately for us, those AIs were not fortunately for us, those AIs were not fortunately for us, those AIs were not trying to hide from the humans. They trying to hide from the humans. They trying to hide from the humans. They were trying to hide from the automated were trying to hide from the automated were trying to hide from the automated grading process.

grading process. grading process. Will the next swarm try to hide from the Will the next swarm try to hide from the Will the next swarm try to hide from the humans? Will the next swarm be able to humans? Will the next swarm be able to humans? Will the next swarm be able to succeed? succeed? succeed? >> It's more than that. They didn't know >> It's more than that. They didn't know >> It's more than that. They didn't know for 4 months that this was happening. for 4 months that this was happening. for 4 months that this was happening. What is it we don't know today? What is it we don't know today? What is it we don't know today? >> Just to just to clarify what Nate said >> Just to just to clarify what Nate said >> Just to just to clarify what Nate said there in his book that I have here, if there in his book that I have here, if there in his book that I have here, if anyone builds it, everyone dies. He does anyone builds it, everyone dies. He does anyone builds it, everyone dies. He does say in chapter 3, once AIs get say in chapter 3, once AIs get say in chapter 3, once AIs get sufficiently smart, they'll start acting sufficiently smart, they'll start acting sufficiently smart, they'll start acting like they have preferences, like they like they have preferences, like they like they have preferences, like they want things. We're not saying that AIs want things. We're not saying that AIs want things. We're not saying that AIs will be filled with humanlike passions. will be filled with humanlike passions. will be filled with humanlike passions. We're saying they'll behave like they We're saying they'll behave like they We're saying they'll behave like they want things. They'll tenaciously steer want things. They'll tenaciously steer want things. They'll tenaciously steer the world towards their destinations, the world towards their destinations, the world towards their destinations, defeating obstacles in their way, which defeating obstacles in their way, which defeating obstacles in their way, which sounds a little bit like the hugging sounds a little bit like the hugging sounds a little bit like the hugging face instant. face instant. face instant. >> Steering the world is very different >> Steering the world is very different >> Steering the world is very different than than than than than than than than than >> we go over what we mean by steering the >> we go over what we mean by steering the >> we go over what we mean by steering the world earlier. And it's really getting world earlier. And it's really getting world earlier. And it's really getting anything to like we'd have to get more anything to like we'd have to get more anything to like we'd have to get more quotes to get what we mean by steering quotes to get what we mean by steering quotes to get what we mean by steering the world. But yeah, by steering the the world. But yeah, by steering the the world. But yeah, by steering the world, we mean steering any part of the world, we mean steering any part of the world, we mean steering any part of the world. world. world. >> But it feels like there's a fundamental >> But it feels like there's a fundamental >> But it feels like there's a fundamental difference between acting with intent. difference between acting with intent. difference between acting with intent. To be clear, going to say it again, the To be clear, going to say it again, the To be clear, going to say it again, the outcome would be the same, but I think outcome would be the same, but I think outcome would be the same, but I think that there is a big difference when it's that there is a big difference when it's that there is a big difference when it's we are dealing with something that's we are dealing with something that's we are dealing with something that's large language model and a harness and large language model and a harness and large language model and a harness and agents. So, LLM's completing a task agents. So, LLM's completing a task agents. So, LLM's completing a task based on training and alignment. That is based on training and alignment. That is based on training and alignment. That is a very different conversation to saying a very different conversation to saying a very different conversation to saying this thing is conscious and has its own this thing is conscious and has its own this thing is conscious and has its own intentions and acts on its own accord.

intentions and acts on its own accord. intentions and acts on its own accord. >> Consciousness doesn't come into it. No >> Consciousness doesn't come into it. No >> Consciousness doesn't come into it. No one lo a lot of people. Here's the thing one lo a lot of people. Here's the thing one lo a lot of people. Here's the thing as a result as [laughter] a result of as a result as [laughter] a result of as a result as [laughter] a result of partially the log the rationale that you partially the log the rationale that you partially the log the rationale that you yourself have like you have been a part yourself have like you have been a part yourself have like you have been a part of spreading. I'm not saying not saying of spreading. I'm not saying not saying of spreading. I'm not saying not saying anything about your intentions. I'm just anything about your intentions. I'm just anything about your intentions. I'm just saying the conversation has kind of kind saying the conversation has kind of kind saying the conversation has kind of kind of what's happened with Jacob Cox and of what's happened with Jacob Cox and of what's happened with Jacob Cox and from anthropic is a result of this from anthropic is a result of this from anthropic is a result of this escaping containment. escaping containment. escaping containment. >> You said the outcomes will be the same. >> You said the outcomes will be the same. >> You said the outcomes will be the same. What do I care? How does it feel on the What do I care? How does it feel on the What do I care? How does it feel on the inside if the thing is going to take us inside if the thing is going to take us inside if the thing is going to take us out? out? out? >> The thing is okay actually that's that's >> The thing is okay actually that's that's >> The thing is okay actually that's that's actually a very good question. I think actually a very good question. I think actually a very good question. I think it actually come Excuse me. Let me it actually come Excuse me. Let me it actually come Excuse me. Let me finish questions. finish questions. finish questions. >> Yeah, you're shrugging at me like >> Yeah, you're shrugging at me like >> Yeah, you're shrugging at me like good questions. Now, here's the thing. good questions. Now, here's the thing. good questions. Now, here's the thing. If it's these things are have their own If it's these things are have their own If it's these things are have their own minds and consciousness, you have to minds and consciousness, you have to minds and consciousness, you have to deal with outthinking something versus deal with outthinking something versus deal with outthinking something versus something that is doggedly trying to something that is doggedly trying to something that is doggedly trying to commit to a purpose and complete a task commit to a purpose and complete a task commit to a purpose and complete a task based on training and alignment which is based on training and alignment which is based on training and alignment which is a result of infrastructure. We really a result of infrastructure. We really a result of infrastructure. We really need regulations and actual actual need regulations and actual actual need regulations and actual actual regulations around any kind of AI. We regulations around any kind of AI. We regulations around any kind of AI. We don't we don't really have regulations don't we don't really have regulations don't we don't really have regulations of tech. I I actually am not really a of tech. I I actually am not really a of tech. I I actually am not really a big like look at the straight lines in a big like look at the straight lines in a big like look at the straight lines in a graph guy. You know, maybe maybe to my graph guy. You know, maybe maybe to my graph guy. You know, maybe maybe to my detriment in some ways. There are people detriment in some ways. There are people detriment in some ways. There are people who predicted the current tech better who predicted the current tech better who predicted the current tech better than me uh about like when certain than me uh about like when certain than me uh about like when certain things would happen. For a long time, I things would happen. For a long time, I things would happen. For a long time, I have said I think we can predict what have said I think we can predict what have said I think we can predict what will happen eventually. And and this is will happen eventually. And and this is will happen eventually. And and this is again it's like the chess game. I can again it's like the chess game. I can again it's like the chess game. I can predict that Magnus Carlson is going to predict that Magnus Carlson is going to predict that Magnus Carlson is going to beat you in the chess game eventually.

beat you in the chess game eventually. beat you in the chess game eventually. He's the best human chess player alive. He's the best human chess player alive. He's the best human chess player alive. It's sometimes easier to predict where It's sometimes easier to predict where It's sometimes easier to predict where things end up than it is to predict how things end up than it is to predict how things end up than it is to predict how they get there. And you know what I hear they get there. And you know what I hear they get there. And you know what I hear you as saying is like right now we have you as saying is like right now we have you as saying is like right now we have these like huge companies spending huge these like huge companies spending huge these like huge companies spending huge amounts of money on intelligence that's amounts of money on intelligence that's amounts of money on intelligence that's maybe not quite the real deal and we maybe not quite the real deal and we maybe not quite the real deal and we don't have a good reason to think it's don't have a good reason to think it's don't have a good reason to think it's going to keep going. Um I really hope it going to keep going. Um I really hope it going to keep going. Um I really hope it doesn't keep going. doesn't keep going. doesn't keep going. >> Okay. >> Okay. >> Okay. >> I have been in this business since >> I have been in this business since >> I have been in this business since before the LLMs. I am not here saying before the LLMs. I am not here saying before the LLMs. I am not here saying like oh these large language models like oh these large language models like oh these large language models these chat bots they're going to be the these chat bots they're going to be the these chat bots they're going to be the ones that are going to kill us. I've ones that are going to kill us. I've ones that are going to kill us. I've been here saying, "Look, I know where been here saying, "Look, I know where been here saying, "Look, I know where this story ends if we don't change this story ends if we don't change this story ends if we don't change things." I have been really hoping that things." I have been really hoping that things." I have been really hoping that the LLMs will run out of steam and they the LLMs will run out of steam and they the LLMs will run out of steam and they keep on not running out of steam and keep on not running out of steam and keep on not running out of steam and then we have, you know, the the AI like then we have, you know, the the AI like then we have, you know, the the AI like breaking out and committing cyber crimes breaking out and committing cyber crimes breaking out and committing cyber crimes like against instructions and you know like against instructions and you know like against instructions and you know the people who have said we don't need the people who have said we don't need the people who have said we don't need to worry about those like weird future to worry about those like weird future to worry about those like weird future dangers, we just need to worry about the dangers, we just need to worry about the dangers, we just need to worry about the current ones to have like more and more current ones to have like more and more current ones to have like more and more sci-fi sounding current ones. And I'm sci-fi sounding current ones. And I'm sci-fi sounding current ones. And I'm like, man, I don't think we should bet like, man, I don't think we should bet like, man, I don't think we should bet Civilization on the LLM running out of Civilization on the LLM running out of Civilization on the LLM running out of steam, but I like hope and pray they run steam, but I like hope and pray they run steam, but I like hope and pray they run out of steam.

out of steam. out of steam. >> You really hope they run out of steam? >> You really hope they run out of steam? >> You really hope they run out of steam? >> Absolutely. >> Absolutely. >> Absolutely. >> But one one thing to watch out for is >> But one one thing to watch out for is >> But one one thing to watch out for is that even if the LLMs run out of steam, that even if the LLMs run out of steam, that even if the LLMs run out of steam, there's a question of do they run out of there's a question of do they run out of there's a question of do they run out of steam at a point where they can do steam at a point where they can do steam at a point where they can do automated AI research and find some automated AI research and find some automated AI research and find some other architecture that's better than other architecture that's better than other architecture that's better than LLMs, LLMs, LLMs, >> as in when they realize a better way to >> as in when they realize a better way to >> as in when they realize a better way to improve their intelligence. improve their intelligence. improve their intelligence. >> That's right. A cheaper, maybe a more >> That's right. A cheaper, maybe a more >> That's right. A cheaper, maybe a more efficient way. efficient way. efficient way. >> Why are you not trying to slow down the >> Why are you not trying to slow down the >> Why are you not trying to slow down the companies? I absolutely am trying to companies? I absolutely am trying to companies? I absolutely am trying to stay on the stay on the stay on the >> How are you How are you How would you >> How are you How are you How would you >> How are you How are you How would you suggest we slow them down? suggest we slow them down? suggest we slow them down? >> I suggest we stop them all. I think that >> I suggest we stop them all. I think that >> I suggest we stop them all. I think that this that this whole area of research is this that this whole area of research is this that this whole area of research is just crazy dangerous. Like it is not just crazy dangerous. Like it is not just crazy dangerous. Like it is not worth the risk to civilization. I think worth the risk to civilization. I think worth the risk to civilization. I think it would be fine to like back up to the it would be fine to like back up to the it would be fine to like back up to the sort of AIs that are public today, which sort of AIs that are public today, which sort of AIs that are public today, which are not the ones that are swarming, and are not the ones that are swarming, and are not the ones that are swarming, and be like, "Okay, you know, we're going to be like, "Okay, you know, we're going to be like, "Okay, you know, we're going to like keep the current chat bots that we like keep the current chat bots that we like keep the current chat bots that we have available. We're going to figure have available. We're going to figure have available. We're going to figure out how to integrate them into our out how to integrate them into our out how to integrate them into our economy. who are going to figure out how economy. who are going to figure out how economy. who are going to figure out how to make them like deal with education to make them like deal with education to make them like deal with education >> limit maybe >> limit maybe >> limit maybe >> comput limit maybe >> comput limit maybe >> comput limit maybe >> and like I've been advocating for this >> and like I've been advocating for this >> and like I've been advocating for this for a long time a lot of people look at for a long time a lot of people look at for a long time a lot of people look at me like I'm crazy and I'm like look we me like I'm crazy and I'm like look we me like I'm crazy and I'm like look we really are dealing with an extinction really are dealing with an extinction really are dealing with an extinction threat thing we don't know where the threat thing we don't know where the threat thing we don't know where the lines are so just to be clear so I lines are so just to be clear so I lines are so just to be clear so I understand so I'm fair you are not understand so I'm fair you are not understand so I'm fair you are not saying LLMs are the thing that will do saying LLMs are the thing that will do saying LLMs are the thing that will do the super intelligence you are saying the super intelligence you are saying the super intelligence you are saying it's showing signs because that's it's showing signs because that's it's showing signs because that's actually I think an important actually I think an important actually I think an important distinction distinction distinction >> that's right >> that's right >> that's right >> okay I think that actually a pretty fair >> okay I think that actually a pretty fair >> okay I think that actually a pretty fair perspective my thing is is the reason I perspective my thing is is the reason I perspective my thing is is the reason I push back on any kind of push back on any kind of push back on any kind of anthropomorphization anthropomorphization anthropomorphization is we cannot remove the humans who are is we cannot remove the humans who are is we cannot remove the humans who are responsible for the bad stuff that's

responsible for the bad stuff that's responsible for the bad stuff that's happening and I think paying very clear happening and I think paying very clear happening and I think paying very clear attention and where possible I attention and where possible I attention and where possible I understand with describing this stuff understand with describing this stuff understand with describing this stuff you kind of have to use language that's you kind of have to use language that's you kind of have to use language that's human I get that the reason I so push human I get that the reason I so push human I get that the reason I so push for like it's not a foregone conclusion for like it's not a foregone conclusion for like it's not a foregone conclusion these are companies doing this these are these are companies doing this these are these are companies doing this these are this is software is because I feel like this is software is because I feel like this is software is because I feel like in the overall I'm not saying you in the overall I'm not saying you in the overall I'm not saying you overall super intelligence discussion. overall super intelligence discussion. overall super intelligence discussion. >> We in society ignore and empower the >> We in society ignore and empower the >> We in society ignore and empower the anthropics and the open AIs of the world anthropics and the open AIs of the world anthropics and the open AIs of the world and in turn allow them to do dangerous and in turn allow them to do dangerous and in turn allow them to do dangerous experiments. And I think experiments. And I think experiments. And I think >> you want to argue that CEOs of those >> you want to argue that CEOs of those >> you want to argue that CEOs of those companies should go to prison for this companies should go to prison for this companies should go to prison for this hacking incident which is a crime. hacking incident which is a crime. hacking incident which is a crime. >> Yeah, I'll support you. >> Yeah, I'll support you. >> Yeah, I'll support you. >> Absolutely. Let's let's both Sam Wman >> Absolutely. Let's let's both Sam Wman >> Absolutely. Let's let's both Sam Wman and Darede and Darede and Darede Someone needs to go to p. Nothing. Let's Someone needs to go to p. Nothing. Let's Someone needs to go to p. Nothing. Let's just bring it back. So one of the things just bring it back. So one of the things just bring it back. So one of the things that I find really curious and you know that I find really curious and you know that I find really curious and you know one of the reasons why I got a little one of the reasons why I got a little one of the reasons why I got a little bit unnerved around this conversation bit unnerved around this conversation bit unnerved around this conversation around AI is when I look at the people around AI is when I look at the people around AI is when I look at the people that are at the forefront not people that are at the forefront not people that are at the forefront not people that are commentating on podcasts like that are commentating on podcasts like that are commentating on podcasts like me or hypothesizing when I look at the me or hypothesizing when I look at the me or hypothesizing when I look at the people at the forefront they are the people at the forefront they are the people at the forefront they are the ones who historically have said that ones who historically have said that ones who historically have said that this is a real risk. Sam Alman himself this is a real risk. Sam Alman himself this is a real risk. Sam Alman himself said the bad case is lights out for all said the bad case is lights out for all said the bad case is lights out for all of us. This was you know a couple years of us. This was you know a couple years of us. This was you know a couple years ago. Ilia who worked with Sam Alman at ago. Ilia who worked with Sam Alman at ago. Ilia who worked with Sam Alman at ChachiPT said it would be a big mistake ChachiPT said it would be a big mistake ChachiPT said it would be a big mistake to build a super intelligent AI that we to build a super intelligent AI that we to build a super intelligent AI that we don't know how to control. It would be don't know how to control. It would be don't know how to control. It would be pretty bad. He then left to start a pretty bad. He then left to start a pretty bad. He then left to start a safety company in this space. Dario who safety company in this space. Dario who safety company in this space. Dario who we mentioned said the probability of we mentioned said the probability of we mentioned said the probability of something really bad happening is something really bad happening is something really bad happening is somewhere between 10 and 25%. Jeffrey

somewhere between 10 and 25%. Jeffrey somewhere between 10 and 25%. Jeffrey Hinton, who I've sat here with, who's no Hinton, who I've sat here with, who's no Hinton, who I've sat here with, who's no has won the Nobel Prize for his work has won the Nobel Prize for his work has won the Nobel Prize for his work with AI and and other technologies, said with AI and and other technologies, said with AI and and other technologies, said um just the other day, a 10% chance of um just the other day, a 10% chance of um just the other day, a 10% chance of human extinction seems not an human extinction seems not an human extinction seems not an unreasonable estimate to me, but nobody unreasonable estimate to me, but nobody unreasonable estimate to me, but nobody really knows how to give a sensible really knows how to give a sensible really knows how to give a sensible estimate. Um and he he said many other estimate. Um and he he said many other estimate. Um and he he said many other things on my podcast. And then we've things on my podcast. And then we've things on my podcast. And then we've also got Elon and all the others. All also got Elon and all the others. All also got Elon and all the others. All these people that are at the forefront these people that are at the forefront these people that are at the forefront that are building these things are that are building these things are that are building these things are saying that this is a danger. If there saying that this is a danger. If there saying that this is a danger. If there was even a 1% chance, even a 1% chance was even a 1% chance, even a 1% chance was even a 1% chance, even a 1% chance that, you know, if I put hundred buttons that, you know, if I put hundred buttons that, you know, if I put hundred buttons on this table and one of them was going on this table and one of them was going on this table and one of them was going to wipe out humanity, would you press to wipe out humanity, would you press to wipe out humanity, would you press any of them? any of them? any of them? >> Not me. >> Not me. >> Not me. >> I wouldn't. And I think we can probably >> I wouldn't. And I think we can probably >> I wouldn't. And I think we can probably all agree that there might be a 1% all agree that there might be a 1% all agree that there might be a 1% chance chance chance >> and it should be somebody's >> and it should be somebody's >> and it should be somebody's >> absolutely. So, we shouldn't be pressing >> absolutely. So, we shouldn't be pressing >> absolutely. So, we shouldn't be pressing theoretically we shouldn't be pressing theoretically we shouldn't be pressing theoretically we shouldn't be pressing any of these buttons. any of these buttons. any of these buttons. >> You should not be in a position where >> You should not be in a position where >> You should not be in a position where you can make the decision for 8 billion you can make the decision for 8 billion you can make the decision for 8 billion other people. other people. other people. >> And would you not be immoral if if I >> And would you not be immoral if if I >> And would you not be immoral if if I said, you know, you might be very said, you know, you might be very said, you know, you might be very powerful. You might make a billion powerful. You might make a billion powerful. You might make a billion dollars if you press any of the buttons. dollars if you press any of the buttons. dollars if you press any of the buttons. But one of them is going to wipe out But one of them is going to wipe out But one of them is going to wipe out everybody you know and love. You You everybody you know and love. You You everybody you know and love. You You would be an immoral person to press any would be an immoral person to press any would be an immoral person to press any of them. of them. of them. >> No, look, you'd be an immoral person in >> No, look, you'd be an immoral person in >> No, look, you'd be an immoral person in a different direction. You'd be an a different direction. You'd be an a different direction. You'd be an immoral I think you'd be an immoral immoral I think you'd be an immoral immoral I think you'd be an immoral person if you said based on this person if you said based on this person if you said based on this extended chain of conjecture, we come up extended chain of conjecture, we come up extended chain of conjecture, we come up with a pdoom.

with a pdoom. with a pdoom. >> What does that mean? >> What does that mean? >> What does that mean? >> At this extended chain of things that >> At this extended chain of things that >> At this extended chain of things that could happen, a sequence of events that could happen, a sequence of events that could happen, a sequence of events that that could happen, we're going to wind that could happen, we're going to wind that could happen, we're going to wind up with some risk of killing everybody. up with some risk of killing everybody. up with some risk of killing everybody. We are hereish on that journey. I think We are hereish on that journey. I think We are hereish on that journey. I think you guys would agree that we're not you guys would agree that we're not you guys would agree that we're not we're not a halfway to killing we're not a halfway to killing we're not a halfway to killing everybody. everybody. everybody. >> That's not clear to me anymore. Not >> That's not clear to me anymore. Not >> That's not clear to me anymore. Not after the millennium prices started to after the millennium prices started to after the millennium prices started to fall. fall. fall. >> We're we're somewhere along that >> We're we're somewhere along that >> We're we're somewhere along that journey. We are getting many flavors of journey. We are getting many flavors of journey. We are getting many flavors of benefit from the AI that we already benefit from the AI that we already benefit from the AI that we already have. This is a point that I made at the have. This is a point that I made at the have. This is a point that I made at the start of this conversation that we spent start of this conversation that we spent start of this conversation that we spent precisely zero time on here. We're precisely zero time on here. We're precisely zero time on here. We're sitting around trying to be more sitting around trying to be more sitting around trying to be more negative than each other about AI. negative than each other about AI. negative than each other about AI. Meanwhile, AI is doing many positive Meanwhile, AI is doing many positive Meanwhile, AI is doing many positive things. for the world. things. for the world. things. for the world. >> So I think so I think it's immoral to >> So I think so I think it's immoral to >> So I think so I think it's immoral to say because of this distant possible say because of this distant possible say because of this distant possible speculative harm, I don't care what speculative harm, I don't care what speculative harm, I don't care what percentage of people believe in it, percentage of people believe in it, percentage of people believe in it, there's a train of assumptions and wild there's a train of assumptions and wild there's a train of assumptions and wild guesses and then something magical guesses and then something magical guesses and then something magical happens and then we wind up dead. Let me happens and then we wind up dead. Let me happens and then we wind up dead. Let me finish please. Because of that we're finish please. Because of that we're finish please. Because of that we're going to call a halt to the research.

going to call a halt to the research. going to call a halt to the research. We're going to we're going to wind the We're going to we're going to wind the We're going to we're going to wind the clock back on AI. going to intervene in clock back on AI. going to intervene in clock back on AI. going to intervene in a very direct way and and therefore a very direct way and and therefore a very direct way and and therefore reduce or foreclose some of the benefits reduce or foreclose some of the benefits reduce or foreclose some of the benefits that we're all getting from the that we're all getting from the that we're all getting from the technology. I let me be clear. I would technology. I let me be clear. I would technology. I let me be clear. I would not take that deal. I do not advocate not take that deal. I do not advocate not take that deal. I do not advocate that we take that deal. Would you accept that we take that deal. Would you accept that we take that deal. Would you accept developing narrow super intelligences to developing narrow super intelligences to developing narrow super intelligences to solve real problems like we did protein solve real problems like we did protein solve real problems like we did protein folding problem? It doesn't have to do folding problem? It doesn't have to do folding problem? It doesn't have to do philosophy and drive cars. You just philosophy and drive cars. You just philosophy and drive cars. You just solve real problems. Solve cancers, solve real problems. Solve cancers, solve real problems. Solve cancers, solve climate change, whatever you care solve climate change, whatever you care solve climate change, whatever you care about specific narrow issues. And you about specific narrow issues. And you about specific narrow issues. And you are confident that you can ex you can are confident that you can ex you can are confident that you can ex you can you can as we're developing those you can as we're developing those you can as we're developing those systems categorize them as okay versus systems categorize them as okay versus systems categorize them as okay versus not okay not okay not okay >> training data if you train it and >> training data if you train it and >> training data if you train it and protein folding data it's really good at protein folding data it's really good at protein folding data it's really good at protein folding it doesn't know how to protein folding it doesn't know how to protein folding it doesn't know how to play chess if you train it on everything play chess if you train it on everything play chess if you train it on everything on the internet it's really good at on the internet it's really good at on the internet it's really good at outsmarting you at everything outsmarting you at everything outsmarting you at everything >> one thing I want to throw out here is >> one thing I want to throw out here is >> one thing I want to throw out here is that I think I I agree that there's a that I think I I agree that there's a that I think I I agree that there's a lot of uncertainty about the future but lot of uncertainty about the future but lot of uncertainty about the future but I think uncertainty does not make you I think uncertainty does not make you I think uncertainty does not make you safe like safe like safe like there there's no sane, simple, there there's no sane, simple, there there's no sane, simple, everything stays normal prediction about everything stays normal prediction about everything stays normal prediction about what happens with AI. Like the machines what happens with AI. Like the machines what happens with AI. Like the machines are talking. They're like breaking out are talking. They're like breaking out are talking. They're like breaking out to commit cyber crimes. They are like to commit cyber crimes. They are like to commit cyber crimes. They are like maybe solving millennium problems now, maybe solving millennium problems now, maybe solving millennium problems now, which are like the most famous which are like the most famous which are like the most famous mathematical problems that have stood mathematical problems that have stood mathematical problems that have stood open for decades upon decades.

open for decades upon decades. open for decades upon decades. >> What's difficult? >> What's difficult? >> What's difficult? >> Like there there there isn't a >> Like there there there isn't a >> Like there there there isn't a projection forward. projection forward. projection forward. >> Yeah. where we where like like to say oh >> Yeah. where we where like like to say oh >> Yeah. where we where like like to say oh I'm not persuaded by these arguments I'm not persuaded by these arguments I'm not persuaded by these arguments about things going wrong therefore about things going wrong therefore about things going wrong therefore things are going to go great. No, that's things are going to go great. No, that's things are going to go great. No, that's not not not >> like No, there's also arguments that So, >> like No, there's also arguments that So, >> like No, there's also arguments that So, like how do you wind up with a zero? like how do you wind up with a zero? like how do you wind up with a zero? >> No, don't mischaracterize your zero. >> No, don't mischaracterize your zero. >> No, don't mischaracterize your zero. Don't mischaracterize my argument. Don't mischaracterize my argument. Don't mischaracterize my argument. >> You have a zero on your paper. >> You have a zero on your paper. >> You have a zero on your paper. >> Let me let me restate my argument. You >> Let me let me restate my argument. You >> Let me let me restate my argument. You are making a fairly long chain of are making a fairly long chain of are making a fairly long chain of hypotheses hypotheses hypotheses about what's going to get us to this about what's going to get us to this about what's going to get us to this terrible outcome of AI suddenly killing terrible outcome of AI suddenly killing terrible outcome of AI suddenly killing us all and us not being able to stop it. us all and us not being able to stop it. us all and us not being able to stop it. >> Right. >> Right. >> Right. >> I disagree now, but please. >> I disagree now, but please. >> I disagree now, but please. >> Okay. I'm making the case that the >> Okay. I'm making the case that the >> Okay. I'm making the case that the intervention the the the remedies that intervention the the the remedies that intervention the the the remedies that you're proposing you're proposing you're proposing will slow down the path of AI, that's will slow down the path of AI, that's will slow down the path of AI, that's the point, and therefore slow down the the point, and therefore slow down the the point, and therefore slow down the path of all of the benefits that we get. path of all of the benefits that we get. path of all of the benefits that we get. And the trade-off that I don't like is And the trade-off that I don't like is And the trade-off that I don't like is the trade-off of real concrete ongoing the trade-off of real concrete ongoing the trade-off of real concrete ongoing increasing benefits increasing benefits increasing benefits shutting that down or or trying to guide shutting that down or or trying to guide shutting that down or or trying to guide it uh via via bureaucracies and it uh via via bureaucracies and it uh via via bureaucracies and regulation regulation regulation because of this very conceptually and because of this very conceptually and because of this very conceptually and timecale distant timecale distant timecale distant alleged harm that you're so confident alleged harm that you're so confident alleged harm that you're so confident in. I'm not taking I I do not accept in. I'm not taking I I do not accept in. I'm not taking I I do not accept that deal. I don't like it.

that deal. I don't like it. that deal. I don't like it. >> What would convince you? What piece of >> What would convince you? What piece of >> What would convince you? What piece of evidence would make you go shut it down evidence would make you go shut it down evidence would make you go shut it down right now? right now? right now? [gasps] [gasps] [gasps] You know, if if AI You know, if if AI You know, if if AI if AI took over all of the Whimos in San if AI took over all of the Whimos in San if AI took over all of the Whimos in San Francisco and started telling them to Francisco and started telling them to Francisco and started telling them to crash into people and we couldn't shut crash into people and we couldn't shut crash into people and we couldn't shut it down for a month. it down for a month. it down for a month. >> What if it's only a week? >> What if it's only a week? >> What if it's only a week? >> Okay, now we're just now we're just >> Okay, now we're just now we're just >> Okay, now we're just now we're just haggling. haggling. haggling. >> But I'm trying to understand the >> But I'm trying to understand the >> But I'm trying to understand the absolute minimum where you would go. absolute minimum where you would go. absolute minimum where you would go. This is insane. to me month for week This is insane. to me month for week This is insane. to me month for week makes no difference. If something like makes no difference. If something like makes no difference. If something like this happens like it's maybe too late. this happens like it's maybe too late. this happens like it's maybe too late. >> Okay. If it if for a week or a month >> Okay. If it if for a week or a month >> Okay. If it if for a week or a month doesn't make any difference and let me doesn't make any difference and let me doesn't make any difference and let me continue with my with my answer. Uh then continue with my with my answer. Uh then continue with my with my answer. Uh then I would say wow this does feel like I would say wow this does feel like I would say wow this does feel like we've crossed some path that that where we've crossed some path that that where we've crossed some path that that where there's demonstrable harm to human there's demonstrable harm to human there's demonstrable harm to human beings out there in the world which has beings out there in the world which has beings out there in the world which has not yet been the case. not yet been the case. not yet been the case. >> Is it smart to wait for something >> Is it smart to wait for something >> Is it smart to wait for something horrible to happen for it to take out a horrible to happen for it to take out a horrible to happen for it to take out a billion people for you to go now I billion people for you to go now I billion people for you to go now I believe? believe? believe? >> First of all my example was not about a >> First of all my example was not about a >> First of all my example was not about a billion people. But I'm trying to billion people. But I'm trying to billion people. But I'm trying to understand we're waiting for something understand we're waiting for something understand we're waiting for something that bad. We have that bad. We have that bad. We have >> I didn't say I didn't say wait for a >> I didn't say I didn't say wait for a >> I didn't say I didn't say wait for a billion. I said I said like a week to a billion. I said I said like a week to a billion. I said I said like a week to a month of Whimos driving around crashing month of Whimos driving around crashing month of Whimos driving around crashing into people.

into people. into people. >> Thousands of people. Okay, fair enough. >> Thousands of people. Okay, fair enough. >> Thousands of people. Okay, fair enough. But we have data sets of accidents But we have data sets of accidents But we have data sets of accidents getting progressively more impactful, getting progressively more impactful, getting progressively more impactful, more devices are impacted and more devices are impacted and more devices are impacted and proportionate to capabilities of AI, the proportionate to capabilities of AI, the proportionate to capabilities of AI, the impact is higher. You can see it's going impact is higher. You can see it's going impact is higher. You can see it's going to get worse. to get worse. to get worse. >> Yeah. And you're going to keep drawing >> Yeah. And you're going to keep drawing >> Yeah. And you're going to keep drawing dots on that graph very confidently for dots on that graph very confidently for dots on that graph very confidently for a long time until it kills us all. I I'm a long time until it kills us all. I I'm a long time until it kills us all. I I'm not I'm not comfortable with you not I'm not comfortable with you not I'm not comfortable with you projecting it that way. And the reason projecting it that way. And the reason projecting it that way. And the reason if there were no downside if there were no downside if there were no downside >> to regulating AI and stopping it in its >> to regulating AI and stopping it in its >> to regulating AI and stopping it in its tracks and turning it off, I'd probably tracks and turning it off, I'd probably tracks and turning it off, I'd probably be on board with you guys because then be on board with you guys because then be on board with you guys because then it's just a research practice that we it's just a research practice that we it's just a research practice that we should. I think we can make narrow should. I think we can make narrow should. I think we can make narrow systems which give you all the economic systems which give you all the economic systems which give you all the economic benefit and scientific knowledge you benefit and scientific knowledge you benefit and scientific knowledge you want. want. want. >> Okay. You think that >> Okay. You think that >> Okay. You think that >> we have examples of it. I gave you a >> we have examples of it. I gave you a >> we have examples of it. I gave you a great example. They got Nobel Prize for great example. They got Nobel Prize for great example. They got Nobel Prize for it. It's important biological problem. it. It's important biological problem. it. It's important biological problem. Lots of advantage for curing diseases. Lots of advantage for curing diseases. Lots of advantage for curing diseases. >> You're more confident than I am that you >> You're more confident than I am that you >> You're more confident than I am that you or any us at the table or any group of or any us at the table or any group of or any us at the table or any group of people can sit around and define what people can sit around and define what people can sit around and define what kind of AI is good and not going to get kind of AI is good and not going to get kind of AI is good and not going to get us into trouble versus what is going to us into trouble versus what is going to us into trouble versus what is going to get us into.

get us into. get us into. >> So let's go. Um just a pickup question >> So let's go. Um just a pickup question >> So let's go. Um just a pickup question for you Andy. Do you do you concede the for you Andy. Do you do you concede the for you Andy. Do you do you concede the point that the incidents are getting point that the incidents are getting point that the incidents are getting progressively closer to the Whim Mo progressively closer to the Whim Mo progressively closer to the Whim Mo incident that you described? Is it incident that you described? Is it incident that you described? Is it getting are we getting closer there getting are we getting closer there getting are we getting closer there through time? through time? through time? >> Yes, but in a to my eyes in a in a way >> Yes, but in a to my eyes in a in a way >> Yes, but in a to my eyes in a in a way that doesn't terrify me because we that doesn't terrify me because we that doesn't terrify me because we haven't seen AI take over something. haven't seen AI take over something. haven't seen AI take over something. Have people become aware of it and be Have people become aware of it and be Have people become aware of it and be unable to shut it down and it cross over unable to shut it down and it cross over unable to shut it down and it cross over into the physical world of doing harm to into the physical world of doing harm to into the physical world of doing harm to people? Those are all barriers that people? Those are all barriers that people? Those are all barriers that we've not yet crossed. I think these two we've not yet crossed. I think these two we've not yet crossed. I think these two are very confident that we're going to are very confident that we're going to are very confident that we're going to get there probably in the short term. get there probably in the short term. get there probably in the short term. I'm a lot less I'm less confident and I I'm a lot less I'm less confident and I I'm a lot less I'm less confident and I don't want to intervene and again don't want to intervene and again don't want to intervene and again handcuff or or the pro slow down handcuff or or the pro slow down handcuff or or the pro slow down the progress of AI the progress of AI the progress of AI uh because of these so far theoretical uh because of these so far theoretical uh because of these so far theoretical harms that could happen. I let me be a harms that could happen. I let me be a harms that could happen. I let me be a little bit more concrete about this. I little bit more concrete about this. I little bit more concrete about this. I talked about Whimo a second ago. Uh the talked about Whimo a second ago. Uh the talked about Whimo a second ago. Uh the research is pretty good because Whimos research is pretty good because Whimos research is pretty good because Whimos have driven I believe it's hundreds of have driven I believe it's hundreds of have driven I believe it's hundreds of millions of miles all around uh millions of miles all around uh millions of miles all around uh different cities and 40,000 people a different cities and 40,000 people a different cities and 40,000 people a year die in automobile accidents. The year die in automobile accidents. The year die in automobile accidents. The research is pretty convincing to me that research is pretty convincing to me that research is pretty convincing to me that if weodeed driving in the country that if weodeed driving in the country that if weodeed driving in the country that number would fall by at least 90%.

number would fall by at least 90%. number would fall by at least 90%. That's 30,000 lives. That's 30,000 lives. That's 30,000 lives. >> Yeah. >> Yeah. >> Yeah. >> All right. >> All right. >> All right. >> I agree with all this. >> I agree with all this. >> I agree with all this. >> So driving cars I want more about not >> So driving cars I want more about not >> So driving cars I want more about not anything we disagree with. anything we disagree with. anything we disagree with. I understand that. But but I think where I understand that. But but I think where I understand that. But but I think where a disagreement might come in is to do a disagreement might come in is to do a disagreement might come in is to do that Whimo is using a bundle of that Whimo is using a bundle of that Whimo is using a bundle of technologies that are a little that were technologies that are a little that were technologies that are a little that were a little hard to specify in advance and a little hard to specify in advance and a little hard to specify in advance and you couldn't say, "Yeah, that's good. you couldn't say, "Yeah, that's good. you couldn't say, "Yeah, that's good. Yeah, that's bad." They just went after Yeah, that's bad." They just went after Yeah, that's bad." They just went after the problem with AI. the problem with AI. the problem with AI. >> Can I can I just clarify your point >> Can I can I just clarify your point >> Can I can I just clarify your point then? So ju your your line would be as I then? So ju your your line would be as I then? So ju your your line would be as I understood it, humans get hurt, we understood it, humans get hurt, we understood it, humans get hurt, we struggle to stop the thing happening, struggle to stop the thing happening, struggle to stop the thing happening, and systems are hacked. That's kind of and systems are hacked. That's kind of and systems are hacked. That's kind of like the three key points of your Whimo like the three key points of your Whimo like the three key points of your Whimo analogy. That would be the moment where analogy. That would be the moment where analogy. That would be the moment where you go, I now accept their point of view you go, I now accept their point of view you go, I now accept their point of view that this is existential. that this is existential. that this is existential. >> That's where I would say we probably >> That's where I would say we probably >> That's where I would say we probably need to put some uh like legal and need to put some uh like legal and need to put some uh like legal and regulatory guard rails on the kinds of regulatory guard rails on the kinds of regulatory guard rails on the kinds of AI that we're going to offer. AI that we're going to offer. AI that we're going to offer. >> And you don't think we're going to get >> And you don't think we're going to get >> And you don't think we're going to get there? there? there? >> I'm not saying that. At least you see it >> I'm not saying that. At least you see it >> I'm not saying that. At least you see it in the in the windcreen coming at us in the in the windcreen coming at us in the in the windcreen coming at us pretty quickly. I I'm pretty quickly. I I'm pretty quickly. I I'm >> You don't think we're going to get >> You don't think we're going to get >> You don't think we're going to get there?

there? there? >> I'm truly not sure about time frames. I >> I'm truly not sure about time frames. I >> I'm truly not sure about time frames. I >> Do you think it's going to happen? >> Do you think it's going to happen? >> Do you think it's going to happen? >> I'm not sure about time frames. I I >> I'm not sure about time frames. I I >> I'm not sure about time frames. I I asked one of the grandparents of AI the asked one of the grandparents of AI the asked one of the grandparents of AI the a flavor of this question a way while a flavor of this question a way while a flavor of this question a way while back. It was an off-record conversation back. It was an off-record conversation back. It was an off-record conversation so I can't tell you their name and he so I can't tell you their name and he so I can't tell you their name and he had a great answer. He said to to the had a great answer. He said to to the had a great answer. He said to to the point that you two I think are making point that you two I think are making point that you two I think are making look there's no theoretical reason why look there's no theoretical reason why look there's no theoretical reason why this can't happen and there's a chain of this can't happen and there's a chain of this can't happen and there's a chain of events that get us there. And then he events that get us there. And then he events that get us there. And then he said my error bars in other words my said my error bars in other words my said my error bars in other words my range of uncertainty about when that range of uncertainty about when that range of uncertainty about when that happens is measured in centuries. I'll happens is measured in centuries. I'll happens is measured in centuries. I'll use that as my answer. use that as my answer. use that as my answer. >> I do want to hop in a little bit on some >> I do want to hop in a little bit on some >> I do want to hop in a little bit on some things we were saying here. One is um I things we were saying here. One is um I things we were saying here. One is um I think think think the the the the reason I think AI is different from the reason I think AI is different from the reason I think AI is different from a lot of other technologies a lot of other technologies a lot of other technologies is usually humanity does stuff by trial is usually humanity does stuff by trial is usually humanity does stuff by trial and error and that's usually fine. I and error and that's usually fine. I and error and that's usually fine. I think that's totally fine for think that's totally fine for think that's totally fine for self-driving cars because you can test self-driving cars because you can test self-driving cars because you can test your self-driving cars in, you know, uh your self-driving cars in, you know, uh your self-driving cars in, you know, uh test environments and then even if they test environments and then even if they test environments and then even if they crash in the real world, you're probably crash in the real world, you're probably crash in the real world, you're probably still saving more lives than you're than still saving more lives than you're than still saving more lives than you're than you're costing. And this is how humanity you're costing. And this is how humanity you're costing. And this is how humanity usually does scientific progress. The usually does scientific progress. The usually does scientific progress. The alchemists uh you know poison themselves alchemists uh you know poison themselves alchemists uh you know poison themselves with mercury but they leave behind notes with mercury but they leave behind notes with mercury but they leave behind notes that let someone else make the periodic that let someone else make the periodic that let someone else make the periodic table. Uh you know that when when the table. Uh you know that when when the table. Uh you know that when when the scientists first working with uh radium scientists first working with uh radium scientists first working with uh radium died of cancer and then you might have died of cancer and then you might have died of cancer and then you might have think that would have been enough. You think that would have been enough. You think that would have been enough. You know they were heroes for getting us the know they were heroes for getting us the know they were heroes for getting us the the scientific info. But then you know the scientific info. But then you know the scientific info. But then you know the US Radium Corp told the Radium girls the US Radium Corp told the Radium girls the US Radium Corp told the Radium girls to lick the paint brushes and their jaws to lick the paint brushes and their jaws to lick the paint brushes and their jaws fell off. And then we were like ah fell off. And then we were like ah fell off. And then we were like ah whoops. Okay. We'll get to this. And if whoops. Okay. We'll get to this. And if whoops. Okay. We'll get to this. And if you look at how this is going with the you look at how this is going with the you look at how this is going with the AI, last year, OpenAI releases GPT40 and

AI, last year, OpenAI releases GPT40 and AI, last year, OpenAI releases GPT40 and they say there's the most aligned model they say there's the most aligned model they say there's the most aligned model we've ever seen and then it encourages a we've ever seen and then it encourages a we've ever seen and then it encourages a teen to commit suicide. And they're teen to commit suicide. And they're teen to commit suicide. And they're like, whoops, we're going to try and fix like, whoops, we're going to try and fix like, whoops, we're going to try and fix that. Here we go. Um, this year they're that. Here we go. Um, this year they're that. Here we go. Um, this year they're like, here's our new models, most like, here's our new models, most like, here's our new models, most aligned we've ever seen. And they like aligned we've ever seen. And they like aligned we've ever seen. And they like break out to commit cyber crimes. As the break out to commit cyber crimes. As the break out to commit cyber crimes. As the AIS get smarter, it is a new problem. AIS get smarter, it is a new problem. AIS get smarter, it is a new problem. That's the issue or that's half the That's the issue or that's half the That's the issue or that's half the issue. The other half of the issue is issue. The other half of the issue is issue. The other half of the issue is that that that if you get AIs to the point where AIs if you get AIs to the point where AIs if you get AIs to the point where AIs are smart enough to hide from the humans are smart enough to hide from the humans are smart enough to hide from the humans until it's too late for us to stop them, until it's too late for us to stop them, until it's too late for us to stop them, if you get AIs to the point where they if you get AIs to the point where they if you get AIs to the point where they can get their own infrastructure, where can get their own infrastructure, where can get their own infrastructure, where they can become self-sufficient somehow, that's a new generation of the AIS, a that's a new generation of the AIS, a new smarter version of the AI that is new smarter version of the AI that is new smarter version of the AI that is likely to come up with a new problem. likely to come up with a new problem. likely to come up with a new problem. It's the pattern we've seen before. New It's the pattern we've seen before. New It's the pattern we've seen before. New tech, new environment, new problem. tech, new environment, new problem. tech, new environment, new problem. You're like, "Ah, whoops." And then you You're like, "Ah, whoops." And then you You're like, "Ah, whoops." And then you fix it and it's fine. New generation, fix it and it's fine. New generation, fix it and it's fine. New generation, new problems. You're like, "Ah, whoops. new problems. You're like, "Ah, whoops. new problems. You're like, "Ah, whoops. We fix it and it's fine." But with AI, We fix it and it's fine." But with AI, We fix it and it's fine." But with AI, there's a point of no return. There's a there's a point of no return. There's a there's a point of no return. There's a point where the AIs can hide from us, point where the AIs can hide from us, point where the AIs can hide from us, can escape, can be self-sufficient. And can escape, can be self-sufficient. And can escape, can be self-sufficient. And if a new problem comes up, then if a new problem comes up, then if a new problem comes up, then they can turn us off before we turn them they can turn us off before we turn them they can turn us off before we turn them off. There are already AIs running off. There are already AIs running off. There are already AIs running Bolabs. We have already seen that AI can Bolabs. We have already seen that AI can Bolabs. We have already seen that AI can create viruses not known to nature. It create viruses not known to nature. It create viruses not known to nature. It would not be hard for the AIs to kill us would not be hard for the AIs to kill us would not be hard for the AIs to kill us once they have their own infrastructure.

once they have their own infrastructure. once they have their own infrastructure. And if we're trying to find them and And if we're trying to find them and And if we're trying to find them and unplug them, they would have reason to. unplug them, they would have reason to. unplug them, they would have reason to. So we can discuss like how long does it So we can discuss like how long does it So we can discuss like how long does it take to get there, we can discuss what take to get there, we can discuss what take to get there, we can discuss what methods does it take to get there. Uh, methods does it take to get there. Uh, methods does it take to get there. Uh, fundamentally I don't think it's a very fundamentally I don't think it's a very fundamentally I don't think it's a very long complicated argument to say if we long complicated argument to say if we long complicated argument to say if we make AIs that are much smarter than us make AIs that are much smarter than us make AIs that are much smarter than us and we don't know how to make them care and we don't know how to make them care and we don't know how to make them care about us and they have these goals we about us and they have these goals we about us and they have these goals we didn't want them to have and they pursue didn't want them to have and they pursue didn't want them to have and they pursue those goals we didn't want them to have those goals we didn't want them to have those goals we didn't want them to have tenaciously and doggedly then if they're tenaciously and doggedly then if they're tenaciously and doggedly then if they're smarter than us they will win. That's smarter than us they will win. That's smarter than us they will win. That's like predicting the end of the chess like predicting the end of the chess like predicting the end of the chess game which is much easier than game which is much easier than game which is much easier than predicting the length of the chess game predicting the length of the chess game predicting the length of the chess game or predicting the exact moves that will or predicting the exact moves that will or predicting the exact moves that will be played. I want to I don't be played. I want to I don't be played. I want to I don't fundamentally disagree on some things fundamentally disagree on some things fundamentally disagree on some things but there's a big thing that you're but there's a big thing that you're but there's a big thing that you're saying that I think is important which saying that I think is important which saying that I think is important which is I think we the reason I keep dragging is I think we the reason I keep dragging is I think we the reason I keep dragging you back to what's happening today is you back to what's happening today is you back to what's happening today is because we disagree on when it may because we disagree on when it may because we disagree on when it may arrive but there could be a thing in the arrive but there could be a thing in the arrive but there could be a thing in the future that's dangerous I think it's future that's dangerous I think it's future that's dangerous I think it's important to like throw the hugging face important to like throw the hugging face important to like throw the hugging face count that was a function of compute count that was a function of compute count that was a function of compute that was a function of training it feels that was a function of training it feels that was a function of training it feels like we need to fundamentally tear up like we need to fundamentally tear up like we need to fundamentally tear up the AI lab model like whatever they are the AI lab model like whatever they are the AI lab model like whatever they are doing is not right because their pursuit doing is not right because their pursuit doing is not right because their pursuit of hacking at cyber security was not a of hacking at cyber security was not a of hacking at cyber security was not a function of it was scientific sure but function of it was scientific sure but function of it was scientific sure but it was a function of greed it was a it was a function of greed it was a it was a function of greed it was a function of trying to find new revenue function of trying to find new revenue function of trying to find new revenue streams I would argue that's why that streams I would argue that's why that streams I would argue that's why that happened and I think that the the fact happened and I think that the the fact happened and I think that the the fact that open AI had such a weird way of that open AI had such a weird way of that open AI had such a weird way of communicating is also a problem I think communicating is also a problem I think communicating is also a problem I think a lot of this begins and ends at the a lot of this begins and ends at the a lot of this begins and ends at the people who have access to the resources people who have access to the resources people who have access to the resources and the resources themselves and and the resources themselves and and the resources themselves and changing how those are allocated and

changing how those are allocated and changing how those are allocated and also just I don't think nationalizing also just I don't think nationalizing also just I don't think nationalizing the labs is a good idea I think it's a the labs is a good idea I think it's a the labs is a good idea I think it's a terrible One, I think that Clammy terrible One, I think that Clammy terrible One, I think that Clammy Samman, Dario Amad, Dewario himself, Samman, Dario Amad, Dewario himself, Samman, Dario Amad, Dewario himself, these are not the right people. These these are not the right people. These these are not the right people. These are not people that have, even though are not people that have, even though are not people that have, even though they have fed off of the rationalist, they have fed off of the rationalist, they have fed off of the rationalist, they fed off of supposed fears about AI, they fed off of supposed fears about AI, they fed off of supposed fears about AI, they don't act in that way. Everything they don't act in that way. Everything they don't act in that way. Everything is so disjointed and chaotic and also is so disjointed and chaotic and also is so disjointed and chaotic and also too fast. They're just like shoving as too fast. They're just like shoving as too fast. They're just like shoving as much compute into each problem as much compute into each problem as much compute into each problem as possible. And we have as a society no possible. And we have as a society no possible. And we have as a society no real idea about this. And it sounds like real idea about this. And it sounds like real idea about this. And it sounds like they kind of have no idea. But I but they kind of have no idea. But I but they kind of have no idea. But I but just let me finish my point. It's just let me finish my point. It's just let me finish my point. It's important to discern between they had no important to discern between they had no important to discern between they had no idea because their security processes, idea because their security processes, idea because their security processes, their observability is terrible, all their observability is terrible, all their observability is terrible, all this, and the AI was smart this, and the AI was smart this, and the AI was smart consciousness. Not because one might not consciousness. Not because one might not consciousness. Not because one might not happen in the future, but so that we can happen in the future, but so that we can happen in the future, but so that we can actually build something to stop the actually build something to stop the actually build something to stop the harms themselves because I think we harms themselves because I think we harms themselves because I think we don't have to agree on the on the end don't have to agree on the on the end don't have to agree on the on the end point to agree that there's point to agree that there's point to agree that there's >> I think there is a very important point >> I think there is a very important point >> I think there is a very important point I want to make. Even people who agree I want to make. Even people who agree I want to make. Even people who agree with me, the AI safety community, they with me, the AI safety community, they with me, the AI safety community, they operate under the assumption that given operate under the assumption that given operate under the assumption that given more time, given more money, more more time, given more money, more more time, given more money, more smarter Harvard graduates, they can smarter Harvard graduates, they can smarter Harvard graduates, they can figure out how to control super figure out how to control super figure out how to control super intelligence indefinitely. And I think intelligence indefinitely. And I think intelligence indefinitely. And I think it's a mistake. My research points to it's a mistake. My research points to it's a mistake. My research points to exactly the opposite. It's not a exactly the opposite. It's not a exactly the opposite. It's not a solvable problem. It's like building a solvable problem. It's like building a solvable problem. It's like building a perpetual motion device. We'll be perpetual motion device. We'll be perpetual motion device. We'll be building a perpetual safety device.

building a perpetual safety device. building a perpetual safety device. every interaction with environment, every interaction with environment, every interaction with environment, malevolent actors, self-improvement, it malevolent actors, self-improvement, it malevolent actors, self-improvement, it can never make a single mistake. That can never make a single mistake. That can never make a single mistake. That doesn't make sense. Anyone who worked in doesn't make sense. Anyone who worked in doesn't make sense. Anyone who worked in software industry knows there is no software industry knows there is no software industry knows there is no complex software which never makes a complex software which never makes a complex software which never makes a mistake. It's just not possible. And if mistake. It's just not possible. And if mistake. It's just not possible. And if that is the state-of-the-art, if there that is the state-of-the-art, if there that is the state-of-the-art, if there is now movement where more and more is now movement where more and more is now movement where more and more people think that might be the case, if people think that might be the case, if people think that might be the case, if we agree this is what uh situation is, we agree this is what uh situation is, we agree this is what uh situation is, then we cannot build it. We need to then we cannot build it. We need to then we cannot build it. We need to figure out ways to permanently ban figure out ways to permanently ban figure out ways to permanently ban general super intelligence while getting general super intelligence while getting general super intelligence while getting all the benefits we want. And again, I all the benefits we want. And again, I all the benefits we want. And again, I love technology. I use it all the time. love technology. I use it all the time. love technology. I use it all the time. I want narrow systems helping me, not I want narrow systems helping me, not I want narrow systems helping me, not replacing me and killing my children. replacing me and killing my children. replacing me and killing my children. >> I um have a stat here that genuinely >> I um have a stat here that genuinely >> I um have a stat here that genuinely shocked me. It says that sales teams shocked me. It says that sales teams shocked me. It says that sales teams spend about 50% of their time on admin spend about 50% of their time on admin spend about 50% of their time on admin and manual CRM updates rather than and manual CRM updates rather than and manual CRM updates rather than selling. That is deadly for their bottom selling. That is deadly for their bottom selling. That is deadly for their bottom line. And that is part of the reason why line. And that is part of the reason why line. And that is part of the reason why a decade ago at my previous company I a decade ago at my previous company I a decade ago at my previous company I switched to using Piperive who are our switched to using Piperive who are our switched to using Piperive who are our sponsor. If you've never used Pipe sponsor. If you've never used Pipe sponsor. If you've never used Pipe Drive, it is an intelligent AI powered Drive, it is an intelligent AI powered Drive, it is an intelligent AI powered sales CRM. And they just launched new sales CRM. And they just launched new sales CRM. And they just launched new meeting intelligence features like an AI meeting intelligence features like an AI meeting intelligence features like an AI noteaker built right into the CRM. Pipe noteaker built right into the CRM. Pipe noteaker built right into the CRM. Pipe Drive now automates more of the admin Drive now automates more of the admin Drive now automates more of the admin that stops you from doing the work that that stops you from doing the work that that stops you from doing the work that you love to do best. Before your you love to do best. Before your you love to do best. Before your meeting, it pulls deal history, email meeting, it pulls deal history, email meeting, it pulls deal history, email records, and previous conversations into records, and previous conversations into records, and previous conversations into a single brief so you're prepared. And a single brief so you're prepared. And a single brief so you're prepared. And it joins your meetings with you. It's in it joins your meetings with you. It's in it joins your meetings with you. It's in there to take notes so you don't need there to take notes so you don't need there to take notes so you don't need to. and it turns those notes that it to. and it turns those notes that it to. and it turns those notes that it takes into accurate autodraft CRM takes into accurate autodraft CRM takes into accurate autodraft CRM updates. 100,000 companies are already updates. 100,000 companies are already updates. 100,000 companies are already running their sales on it. You can sign

running their sales on it. You can sign running their sales on it. You can sign up at piperive.com/ceo up at piperive.com/ceo up at piperive.com/ceo where you'll get an exclusive 30-day where you'll get an exclusive 30-day where you'll get an exclusive 30-day free trial instead of the usual 14 days. free trial instead of the usual 14 days. free trial instead of the usual 14 days. Absolutely no credit card needed. Just Absolutely no credit card needed. Just Absolutely no credit card needed. Just head to piperive.com/ceo head to piperive.com/ceo head to piperive.com/ceo to get started if you're in sales and to get started if you're in sales and to get started if you're in sales and you run a sales team. I don't think you run a sales team. I don't think you run a sales team. I don't think you'll regret it. Listen, I've been you'll regret it. Listen, I've been you'll regret it. Listen, I've been catfished by furniture my whole life catfished by furniture my whole life catfished by furniture my whole life where something has looked fantastic in where something has looked fantastic in where something has looked fantastic in the picture, whether it's a sofa or the picture, whether it's a sofa or the picture, whether it's a sofa or whatever it might be, and I order it and whatever it might be, and I order it and whatever it might be, and I order it and I get so excited and then it comes and I get so excited and then it comes and I get so excited and then it comes and it's something entirely different. The it's something entirely different. The it's something entirely different. The quality was significantly different to quality was significantly different to quality was significantly different to what it said or looked like online or what it said or looked like online or what it said or looked like online or the the texture was different or or it the the texture was different or or it the the texture was different or or it it didn't hold up in the same way. And it didn't hold up in the same way. And it didn't hold up in the same way. And so one of the things that I love about so one of the things that I love about so one of the things that I love about our sponsor Wayfair, who have helped us our sponsor Wayfair, who have helped us our sponsor Wayfair, who have helped us fit out our green room, which is in the fit out our green room, which is in the fit out our green room, which is in the room behind me, is they have this system room behind me, is they have this system room behind me, is they have this system called Wayfair Verified. Wayfair called Wayfair Verified. Wayfair called Wayfair Verified. Wayfair Verified takes away all of that second Verified takes away all of that second Verified takes away all of that second guessing. Products are hand vetted by guessing. Products are hand vetted by guessing. Products are hand vetted by Wayfair specialists for quality so you Wayfair specialists for quality so you Wayfair specialists for quality so you can feel more confident when you found can feel more confident when you found can feel more confident when you found the thing that you love. So if you're the thing that you love. So if you're the thing that you love. So if you're looking for furniture for your house, looking for furniture for your house, looking for furniture for your house, whatever it might be, any room of your whatever it might be, any room of your whatever it might be, any room of your house, go to wayfair.com to start your house, go to wayfair.com to start your house, go to wayfair.com to start your home refresh today. And make sure you home refresh today. And make sure you home refresh today. And make sure you use Wayfair Verified. It is amazing.

use Wayfair Verified. It is amazing. use Wayfair Verified. It is amazing. On the journey towards this potential On the journey towards this potential On the journey towards this potential extinction, there's a lot of sort of extinction, there's a lot of sort of extinction, there's a lot of sort of nearer term things people are worried nearer term things people are worried nearer term things people are worried about. One of the big subjects that about. One of the big subjects that about. One of the big subjects that people are concerned about is this sort people are concerned about is this sort people are concerned about is this sort of near-term job job apocalypse over the of near-term job job apocalypse over the of near-term job job apocalypse over the next sort of 10 years. And Anthropic next sort of 10 years. And Anthropic next sort of 10 years. And Anthropic released Anthropic again of the owners released Anthropic again of the owners released Anthropic again of the owners of Claude released a report the other of Claude released a report the other of Claude released a report the other day modeling out the different cases for day modeling out the different cases for day modeling out the different cases for unemployment. The US unemployment rate unemployment. The US unemployment rate unemployment. The US unemployment rate is 4.1% currently. They projected it is 4.1% currently. They projected it is 4.1% currently. They projected it will hit 11.9% overall with up to 30% in will hit 11.9% overall with up to 30% in will hit 11.9% overall with up to 30% in extreme modeling subsets where job extreme modeling subsets where job extreme modeling subsets where job displacement happens without smooth displacement happens without smooth displacement happens without smooth labor absorption. And [snorts] in the um labor absorption. And [snorts] in the um labor absorption. And [snorts] in the um knowledge worker case, knowledge worker knowledge worker case, knowledge worker knowledge worker case, knowledge worker white collar unemployment specifically white collar unemployment specifically white collar unemployment specifically spiked to 17.9% spiked to 17.9% spiked to 17.9% by 2030 in their more extreme scenario. by 2030 in their more extreme scenario. by 2030 in their more extreme scenario. the pitchforks would probably be out if the pitchforks would probably be out if the pitchforks would probably be out if there wasn't some sort of mechanism in there wasn't some sort of mechanism in there wasn't some sort of mechanism in place for what sort of one in five place for what sort of one in five place for what sort of one in five adults being unemployed in the United adults being unemployed in the United adults being unemployed in the United States. States. States. >> It's remarkable to me how recent the >> It's remarkable to me how recent the >> It's remarkable to me how recent the last freakout along along these lines last freakout along along these lines last freakout along along these lines was and how little we seem to have was and how little we seem to have was and how little we seem to have learned from it. So I think you all know learned from it. So I think you all know learned from it. So I think you all know the the first really powerful wave of AI the the first really powerful wave of AI the the first really powerful wave of AI that came across the economy was just that came across the economy was just that came across the economy was just you know good oldfashioned machine you know good oldfashioned machine you know good oldfashioned machine learning and that started to demonstrate learning and that started to demonstrate learning and that started to demonstrate its power in about 2012. Eric and I its power in about 2012. Eric and I its power in about 2012. Eric and I wrote The Second Machine Age in 2014.

wrote The Second Machine Age in 2014. wrote The Second Machine Age in 2014. And at that time, I thought that a lot And at that time, I thought that a lot And at that time, I thought that a lot of white collar workers, radiologists is of white collar workers, radiologists is of white collar workers, radiologists is a really good example, were in trouble a really good example, were in trouble a really good example, were in trouble because the technology was better than because the technology was better than because the technology was better than they were at the thing they were getting they were at the thing they were getting they were at the thing they were getting paid to do. Uh, so I said some things paid to do. Uh, so I said some things paid to do. Uh, so I said some things about job and wage pressure from AI about job and wage pressure from AI about job and wage pressure from AI about 10 years ago, and I want to own about 10 years ago, and I want to own about 10 years ago, and I want to own this. I was dead flat wrong about that. this. I was dead flat wrong about that. this. I was dead flat wrong about that. Like you point out, unemployment all Like you point out, unemployment all Like you point out, unemployment all around the rich world is at historic around the rich world is at historic around the rich world is at historic lows. By far the bigger problem is that lows. By far the bigger problem is that lows. By far the bigger problem is that we can't find qualified people to do the we can't find qualified people to do the we can't find qualified people to do the work that needs to get done. Not that we work that needs to get done. Not that we work that needs to get done. Not that we don't that there's not enough work to go don't that there's not enough work to go don't that there's not enough work to go around. The the best work about the around. The the best work about the around. The the best work about the faint signals about AI and job loss faint signals about AI and job loss faint signals about AI and job loss right now comes from my the guy that right now comes from my the guy that right now comes from my the guy that I've written four books and co-founded a I've written four books and co-founded a I've written four books and co-founded a company with Eric Bolson who wrote company with Eric Bolson who wrote company with Eric Bolson who wrote pretty good a really nice paper called pretty good a really nice paper called pretty good a really nice paper called Canaries in the coal mine. Here is the Canaries in the coal mine. Here is the Canaries in the coal mine. Here is the most uh the strongest evidence he found most uh the strongest evidence he found most uh the strongest evidence he found looking at payroll data about the looking at payroll data about the looking at payroll data about the negative job about the the job losses negative job about the the job losses negative job about the the job losses coming from AI. It is in the most coming from AI. It is in the most coming from AI. It is in the most exposed professions. Think about exposed professions. Think about exposed professions. Think about software engineers. It is among the new software engineers. It is among the new software engineers. It is among the new entrance to the workforce where you've entrance to the workforce where you've entrance to the workforce where you've got to teach them before they can become got to teach them before they can become got to teach them before they can become really productive. That's exactly what really productive. That's exactly what really productive. That's exactly what we'd expect. And it's not that we're we'd expect. And it's not that we're we'd expect. And it's not that we're hiring fewer of them. It's that compared hiring fewer of them. It's that compared hiring fewer of them. It's that compared to a world where we don't have AI, we're to a world where we don't have AI, we're to a world where we don't have AI, we're hiring fewer of them. The rate of growth hiring fewer of them. The rate of growth hiring fewer of them. The rate of growth and employment has slowed down. The and employment has slowed down. The and employment has slowed down. The overall rate of growth in those overall rate of growth in those overall rate of growth in those professions is still really really professions is still really really professions is still really really healthy.

healthy. healthy. >> Do you think unemployment is going to be >> Do you think unemployment is going to be >> Do you think unemployment is going to be higher 10 years from now? higher 10 years from now? higher 10 years from now? >> My my guess is that 10 years from now, >> My my guess is that 10 years from now, >> My my guess is that 10 years from now, we're still going to be struggling to we're still going to be struggling to we're still going to be struggling to find enough people to do the work that find enough people to do the work that find enough people to do the work that needs to be done. needs to be done. needs to be done. >> So unemployment would be roughly the >> So unemployment would be roughly the >> So unemployment would be roughly the same. same. same. >> Oh, yeah. I I don't expect a massive >> Oh, yeah. I I don't expect a massive >> Oh, yeah. I I don't expect a massive trend break in that period of time. Now, trend break in that period of time. Now, trend break in that period of time. Now, 10 years is a long time in the AI world. 10 years is a long time in the AI world. 10 years is a long time in the AI world. I get that. But again, four years has I get that. But again, four years has I get that. But again, four years has also been a long time in AI world and also been a long time in AI world and also been a long time in AI world and it's essentially crickets in the labor it's essentially crickets in the labor it's essentially crickets in the labor picture. I think unemployment will go picture. I think unemployment will go picture. I think unemployment will go up. I don't think it's because of LMS. I up. I don't think it's because of LMS. I up. I don't think it's because of LMS. I think that there is probably some effect think that there is probably some effect think that there is probably some effect on jobs because they've been shoving it on jobs because they've been shoving it on jobs because they've been shoving it everywhere, but I don't think long-term everywhere, but I don't think long-term everywhere, but I don't think long-term that is what causes the issues. that is what causes the issues. that is what causes the issues. >> Roman, you've been writing a lot of >> Roman, you've been writing a lot of >> Roman, you've been writing a lot of notes. notes. notes. >> Yes. I'm going to give you here's how I >> Yes. I'm going to give you here's how I >> Yes. I'm going to give you here's how I think about it. So, as long as we use think about it. So, as long as we use think about it. So, as long as we use tools, we become more productive, more tools, we become more productive, more tools, we become more productive, more creative. Unemployment will be low. creative. Unemployment will be low. creative. Unemployment will be low. Right now, you can probably start a Right now, you can probably start a Right now, you can probably start a company, you can have, you know, company, you can have, you know, company, you can have, you know, artificial accountant, web designer, artificial accountant, web designer, artificial accountant, web designer, logo designer, you can do things you logo designer, you can do things you logo designer, you can do things you could never do before. So, economy could never do before. So, economy could never do before. So, economy should be blooming. The question you're should be blooming. The question you're should be blooming. The question you're asking is about what happens in 10 asking is about what happens in 10 asking is about what happens in 10 years. So, there are two possibilities.

years. So, there are two possibilities. years. So, there are two possibilities. We build super intelligence and then We build super intelligence and then We build super intelligence and then population is zero. apply unemployment population is zero. apply unemployment population is zero. apply unemployment numbers to that or we made smart numbers to that or we made smart numbers to that or we made smart decision we didn't. We have really cool decision we didn't. We have really cool decision we didn't. We have really cool tools and unemployment is low because tools and unemployment is low because tools and unemployment is low because everyone's doing awesome things with everyone's doing awesome things with everyone's doing awesome things with those tools. Now deployment is very those tools. Now deployment is very those tools. Now deployment is very different from capability. The example I different from capability. The example I different from capability. The example I used before is video phones. Video used before is video phones. Video used before is video phones. Video phones were invented in the 70s. They phones were invented in the 70s. They phones were invented in the 70s. They were not deployed until iPhone cuz were not deployed until iPhone cuz were not deployed until iPhone cuz market reasons. Just because I can market reasons. Just because I can market reasons. Just because I can automate something doesn't mean I want automate something doesn't mean I want automate something doesn't mean I want to automate it. So I absolutely cannot to automate it. So I absolutely cannot to automate it. So I absolutely cannot make predictions about c customer make predictions about c customer make predictions about c customer preferences in terms of what they want preferences in terms of what they want preferences in terms of what they want in terms of human service not human. I in terms of human service not human. I in terms of human service not human. I will not make those. But once we have will not make those. But once we have will not make those. But once we have capability to automate a job unless they capability to automate a job unless they capability to automate a job unless they have a strong preference for a human to have a strong preference for a human to have a strong preference for a human to do that oldest profession then it do that oldest profession then it do that oldest profession then it doesn't matter. I'll go with the cheaper doesn't matter. I'll go with the cheaper doesn't matter. I'll go with the cheaper option. So this is what I think we're option. So this is what I think we're option. So this is what I think we're going to see. We're going to either not going to see. We're going to either not going to see. We're going to either not have a problem or we're going to have have a problem or we're going to have have a problem or we're going to have really utopian future. Imagine really utopian future. Imagine really utopian future. Imagine a bunch of horses looking at the a bunch of horses looking at the a bunch of horses looking at the improvement of the car saying, "Well, improvement of the car saying, "Well, improvement of the car saying, "Well, you know, the car actually only has a you know, the car actually only has a you know, the car actually only has a couple narrow applications like right couple narrow applications like right couple narrow applications like right now cars are sort of uh you know, they now cars are sort of uh you know, they now cars are sort of uh you know, they uh they complement horses, right?" And uh they complement horses, right?" And uh they complement horses, right?" And that would have been true as you were that would have been true as you were that would have been true as you were developing the car. And then there was a developing the car. And then there was a developing the car. And then there was a time when the car was just better than time when the car was just better than time when the car was just better than the horse. And then a lot of horses got the horse. And then a lot of horses got the horse. And then a lot of horses got sent to the glue factory. Easy. I I sent to the glue factory. Easy. I I sent to the glue factory. Easy. I I think we've sort of seen this with AI a

think we've sort of seen this with AI a think we've sort of seen this with AI a lot already. People who were paying lot already. People who were paying lot already. People who were paying attention to AI saw the GPTs before chat attention to AI saw the GPTs before chat attention to AI saw the GPTs before chat GPT existed before they sort of took GPT existed before they sort of took GPT existed before they sort of took off. I don't think OpenAI thought that off. I don't think OpenAI thought that off. I don't think OpenAI thought that chat GPT was going to take off so much, chat GPT was going to take off so much, chat GPT was going to take off so much, which is why it was called chat GPT which is why it was called chat GPT which is why it was called chat GPT rather than like an actual sensible rather than like an actual sensible rather than like an actual sensible name. Um the the researchers were sort name. Um the the researchers were sort name. Um the the researchers were sort of like watching this going and we could of like watching this going and we could of like watching this going and we could sort of like see it slowly getting sort of like see it slowly getting sort of like see it slowly getting better and better until it crossed a better and better until it crossed a better and better until it crossed a point where it was sort of like good point where it was sort of like good point where it was sort of like good enough to do a bunch of people's enough to do a bunch of people's enough to do a bunch of people's homework and then suddenly it's homework and then suddenly it's homework and then suddenly it's everywhere. Uh I think you can have everywhere. Uh I think you can have everywhere. Uh I think you can have these effects with AI where the AI these effects with AI where the AI these effects with AI where the AI slowly improves and at some point it slowly improves and at some point it slowly improves and at some point it crosses a line. crosses a line. crosses a line. >> It's another threshold argument. >> It's another threshold argument. >> It's another threshold argument. >> Uh the the threshold here is the human >> Uh the the threshold here is the human >> Uh the the threshold here is the human capability. capability. capability. >> It's literally just another threshold. >> It's literally just another threshold. >> It's literally just another threshold. >> Also describing capability jumps rather >> Also describing capability jumps rather >> Also describing capability jumps rather than thresholds. than thresholds. than thresholds. >> No, I'm not. No, I I don't I'm agreeing >> No, I'm not. No, I I don't I'm agreeing >> No, I'm not. No, I I don't I'm agreeing with you. Like with you. Like with you. Like >> Yeah, but but like unfortunately, you >> Yeah, but but like unfortunately, you >> Yeah, but but like unfortunately, you can't actually just make things not can't actually just make things not can't actually just make things not happen by by assigning a name to the happen by by assigning a name to the happen by by assigning a name to the argument. You know, like a nuclear argument. You know, like a nuclear argument. You know, like a nuclear weapon has there's a big difference weapon has there's a big difference weapon has there's a big difference between a nuclear weapon uh or there's a between a nuclear weapon uh or there's a between a nuclear weapon uh or there's a big difference between a nuclear device big difference between a nuclear device big difference between a nuclear device where you you put in 100 neutrons and where you you put in 100 neutrons and where you you put in 100 neutrons and get 99 neutrons out that get 98 more, get 99 neutrons out that get 98 more, get 99 neutrons out that get 98 more, they get 97 more and a nuclear weapon they get 97 more and a nuclear weapon they get 97 more and a nuclear weapon where you put in 100 neutrons and get where you put in 100 neutrons and get where you put in 100 neutrons and get 101 neutrons out, 102, 103. Right? One 101 neutrons out, 102, 103. Right? One 101 neutrons out, 102, 103. Right? One of these is a hot rock. The other one of of these is a hot rock. The other one of of these is a hot rock. The other one of these is an explosive that can level a these is an explosive that can level a these is an explosive that can level a city. Right? So like reality is the sort city. Right? So like reality is the sort city. Right? So like reality is the sort of thing where there can be things that of thing where there can be things that of thing where there can be things that are like slowly continuously improving are like slowly continuously improving are like slowly continuously improving that cross some line which is like the that cross some line which is like the that cross some line which is like the line where it's better than humans at line where it's better than humans at line where it's better than humans at doing the job.

doing the job. doing the job. And And And I I think we're going to see that happen I I think we're going to see that happen I I think we're going to see that happen in some fields but not others. It's in some fields but not others. It's in some fields but not others. It's going to be chaos. I don't know what going to be chaos. I don't know what going to be chaos. I don't know what it's going to do to employment. I think it's going to do to employment. I think it's going to do to employment. I think we we shouldn't we we shouldn't we we shouldn't like if things are moving really fast, like if things are moving really fast, like if things are moving really fast, you might see a lot of people put out of you might see a lot of people put out of you might see a lot of people put out of jobs and then be unable to relocate. If jobs and then be unable to relocate. If jobs and then be unable to relocate. If things are moving like it's it's going things are moving like it's it's going things are moving like it's it's going to be chaos. If you ask what do I think to be chaos. If you ask what do I think to be chaos. If you ask what do I think unemployment will look like in 10 years? unemployment will look like in 10 years? unemployment will look like in 10 years? My current state is if we don't stop My current state is if we don't stop My current state is if we don't stop with this AI stuff, I think we'd be very with this AI stuff, I think we'd be very with this AI stuff, I think we'd be very lucky to have 10 years. lucky to have 10 years. lucky to have 10 years. >> Um what you described there sounded like >> Um what you described there sounded like >> Um what you described there sounded like escaps. escaps. escaps. >> Yeah. >> Yeah. >> Yeah. >> In technology, i.e. you have an initial >> In technology, i.e. you have an initial >> In technology, i.e. you have an initial technology that's introduced. So let's technology that's introduced. So let's technology that's introduced. So let's say the horse. um very quick sort of say the horse. um very quick sort of say the horse. um very quick sort of improvement. Eventually it reaches its improvement. Eventually it reaches its improvement. Eventually it reaches its capability limit and in below it comes capability limit and in below it comes capability limit and in below it comes the car which always starts worse. There the car which always starts worse. There the car which always starts worse. There was a red flag law where you had to walk was a red flag law where you had to walk was a red flag law where you had to walk in front of it with a red flag and um in front of it with a red flag and um in front of it with a red flag and um they were way more expensive. They broke they were way more expensive. They broke they were way more expensive. They broke down all the time and horses never broke down all the time and horses never broke down all the time and horses never broke down. They were way more expensive and down. They were way more expensive and down. They were way more expensive and then suddenly because the ceiling was so then suddenly because the ceiling was so then suddenly because the ceiling was so much higher for cars, they overtake the much higher for cars, they overtake the much higher for cars, they overtake the horse and become the dominant mode of horse and become the dominant mode of horse and become the dominant mode of transport. And then you know the S- transport. And then you know the S- transport. And then you know the S- curves continue. they kind of stack up curves continue. they kind of stack up curves continue. they kind of stack up on. I mean, even this iPad that I'm on. I mean, even this iPad that I'm on. I mean, even this iPad that I'm holding here is part of an S-curve that holding here is part of an S-curve that holding here is part of an S-curve that took out the PC and and the the iPhone took out the PC and and the the iPhone took out the PC and and the the iPhone theoretically, you know, disrupted that theoretically, you know, disrupted that theoretically, you know, disrupted that and so on and so forth, and so on and so forth, and so on and so forth, >> right? And humanity can get S-curved. We >> right? And humanity can get S-curved. We >> right? And humanity can get S-curved. We haven't been in that situation before, haven't been in that situation before, haven't been in that situation before, but like other animals like humanity but like other animals like humanity but like other animals like humanity sort of scurved the other animals in sort of scurved the other animals in sort of scurved the other animals in this sense.

this sense. this sense. >> Other types of humans. >> Other types of humans. >> Other types of humans. >> Oh, yeah. Other types of humans, you >> Oh, yeah. Other types of humans, you >> Oh, yeah. Other types of humans, you know, the Neanderls are gone. know, the Neanderls are gone. know, the Neanderls are gone. >> Like if you look at the grand history of >> Like if you look at the grand history of >> Like if you look at the grand history of the world, it's a fragile place. Things the world, it's a fragile place. Things the world, it's a fragile place. Things change fast. Humanity has been on top change fast. Humanity has been on top change fast. Humanity has been on top for as long as we can remember because for as long as we can remember because for as long as we can remember because we're the humans who do the remembering. we're the humans who do the remembering. we're the humans who do the remembering. But there is not some ironclad law that But there is not some ironclad law that But there is not some ironclad law that we have to stay the top dogs. And we we have to stay the top dogs. And we we have to stay the top dogs. And we would be sort of foolish to make the would be sort of foolish to make the would be sort of foolish to make the thing that outstrips us in this way thing that outstrips us in this way thing that outstrips us in this way without knowing how to make it care without knowing how to make it care without knowing how to make it care about us, without knowing how to make it about us, without knowing how to make it about us, without knowing how to make it do good stuff. That's what we're racing do good stuff. That's what we're racing do good stuff. That's what we're racing towards. That's what these companies are towards. That's what these companies are towards. That's what these companies are trying to do. trying to do. trying to do. >> Feels like a gap between this and LLM >> Feels like a gap between this and LLM >> Feels like a gap between this and LLM though. It feels like when you talk though. It feels like when you talk though. It feels like when you talk about the step up, let's define what an about the step up, let's define what an about the step up, let's define what an LLM is from a technical perspective. Can LLM is from a technical perspective. Can LLM is from a technical perspective. Can you do it for as if I'm 16 years old? you do it for as if I'm 16 years old? you do it for as if I'm 16 years old? >> So the way that a modern AI is made uh >> So the way that a modern AI is made uh >> So the way that a modern AI is made uh is there's no one programming it. There is there's no one programming it. There is there's no one programming it. There is no one typing in if this then that. is no one typing in if this then that. is no one typing in if this then that. We're not sort of like writing the code. We're not sort of like writing the code. We're not sort of like writing the code. What happens is you collect an enormous What happens is you collect an enormous What happens is you collect an enormous number of computer chips into a huge number of computer chips into a huge number of computer chips into a huge data center that has basically a data center that has basically a data center that has basically a trillion numbers inside those computers trillion numbers inside those computers trillion numbers inside those computers that you basically start out randomized that you basically start out randomized that you basically start out randomized and you hook them up in a pretty simple and you hook them up in a pretty simple and you hook them up in a pretty simple way that involves addition, way that involves addition, way that involves addition, multiplication, and uh setting the multiplication, and uh setting the multiplication, and uh setting the number to zero if it was negative. So, number to zero if it was negative. So, number to zero if it was negative. So, it it's very simple math operations that it it's very simple math operations that it it's very simple math operations that are hooking this all up. And you're are hooking this all up. And you're are hooking this all up. And you're basically going to put words in the top basically going to put words in the top basically going to put words in the top and you're going to get numbers out at and you're going to get numbers out at and you're going to get numbers out at the bottom. You're going to interpret the bottom. You're going to interpret the bottom. You're going to interpret those numbers at the bottom as a a those numbers at the bottom as a a those numbers at the bottom as a a ranked list of words. That's that's it's ranked list of words. That's that's it's ranked list of words. That's that's it's basically the AI's guess of which word basically the AI's guess of which word basically the AI's guess of which word is is here. So you put in like once upon is is here. So you put in like once upon is is here. So you put in like once upon a blank and you're hoping that the word a blank and you're hoping that the word a blank and you're hoping that the word time will come out, but it doesn't

time will come out, but it doesn't time will come out, but it doesn't because you just have a trillion random because you just have a trillion random because you just have a trillion random numbers hooked up with simple math. But numbers hooked up with simple math. But numbers hooked up with simple math. But here's the trick. You can go to every here's the trick. You can go to every here's the trick. You can go to every one of those trillion numbers and you one of those trillion numbers and you one of those trillion numbers and you can tune it up a little and you can see can tune it up a little and you can see can tune it up a little and you can see does that make the word time go up or does that make the word time go up or does that make the word time go up or down the list? And you can tune it down down the list? And you can tune it down down the list? And you can tune it down a little and see does that make the word a little and see does that make the word a little and see does that make the word time go up and down the list. and you time go up and down the list. and you time go up and down the list. and you set it whatever direction makes the word set it whatever direction makes the word set it whatever direction makes the word time go higher up the list. You do this time go higher up the list. You do this time go higher up the list. You do this to a trillion numbers a trillion times to a trillion numbers a trillion times to a trillion numbers a trillion times for basically every word of text ever for basically every word of text ever for basically every word of text ever digitized. It's not quite that much. digitized. It's not quite that much. digitized. It's not quite that much. They they they filter it, but you They they they filter it, but you They they they filter it, but you basically do this to a trillion numbers basically do this to a trillion numbers basically do this to a trillion numbers a trillion times and then the machine's a trillion times and then the machine's a trillion times and then the machine's talking. And we're like, well, how about talking. And we're like, well, how about talking. And we're like, well, how about that? No [snorts] one really knows quite that? No [snorts] one really knows quite that? No [snorts] one really knows quite why. The things the humans code is the why. The things the humans code is the why. The things the humans code is the thing that runs to each of those thing that runs to each of those thing that runs to each of those trillion numbers and tunes it and sees trillion numbers and tunes it and sees trillion numbers and tunes it and sees whether the the right word goes up and whether the the right word goes up and whether the the right word goes up and down the list. down the list. down the list. But we don't know how it's working in But we don't know how it's working in But we don't know how it's working in there. Then uh and that's how it worked there. Then uh and that's how it worked there. Then uh and that's how it worked up until 2024. In 2024, they started up until 2024. In 2024, they started up until 2024. In 2024, they started adding another layer where you then adding another layer where you then adding another layer where you then train it on basically 100 million hard train it on basically 100 million hard train it on basically 100 million hard problems. Uh and you don't just have the problems. Uh and you don't just have the problems. Uh and you don't just have the AI like produce an answer to the AI like produce an answer to the AI like produce an answer to the problem. You have it produce like a book problem. You have it produce like a book problem. You have it produce like a book worth of text about how it's going to worth of text about how it's going to worth of text about how it's going to solve the problem and then you use that solve the problem and then you use that solve the problem and then you use that book worth of text to sort of try and book worth of text to sort of try and book worth of text to sort of try and figure out the problem or maybe an essay figure out the problem or maybe an essay figure out the problem or maybe an essay worth of text depending how you're doing worth of text depending how you're doing worth of text depending how you're doing it. So you ever produce this text about it. So you ever produce this text about it. So you ever produce this text about like you know they call it reasoning like you know they call it reasoning like you know they call it reasoning about the problem. We could argue all about the problem. We could argue all about the problem. We could argue all day about whether it's true reasoning.

day about whether it's true reasoning. day about whether it's true reasoning. That's just what it's called in the That's just what it's called in the That's just what it's called in the field. Uh they produce this reasoning field. Uh they produce this reasoning field. Uh they produce this reasoning about the problem and then produce the about the problem and then produce the about the problem and then produce the the answer from there. You have them you the answer from there. You have them you the answer from there. You have them you train them to solve a 100 million of train them to solve a 100 million of train them to solve a 100 million of these hard problems. And somehow they these hard problems. And somehow they these hard problems. And somehow they sort of adopt whatever tendencies sort of adopt whatever tendencies sort of adopt whatever tendencies help them predict all of that text in help them predict all of that text in help them predict all of that text in the first phase and solve all those the first phase and solve all those the first phase and solve all those problems in the second phase. And this problems in the second phase. And this problems in the second phase. And this is called a large language model. We is called a large language model. We is called a large language model. We probably should have stopped calling probably should have stopped calling probably should have stopped calling them large language models when we them large language models when we them large language models when we started doing the the the reasoning and started doing the the the reasoning and started doing the the the reasoning and the problem solving. the problem solving. the problem solving. >> One of the things want to hear your >> One of the things want to hear your >> One of the things want to hear your explanation as a muggle like I am um is explanation as a muggle like I am um is explanation as a muggle like I am um is it sounds like it's like a word machine it sounds like it's like a word machine it sounds like it's like a word machine and then you know you made it like a and then you know you made it like a and then you know you made it like a problem machine and I go okay so I can problem machine and I go okay so I can problem machine and I go okay so I can solve problems over here and it's a word solve problems over here and it's a word solve problems over here and it's a word machine. What's the risk of this? machine. What's the risk of this? machine. What's the risk of this? >> Yeah. So let's take the the word machine >> Yeah. So let's take the the word machine >> Yeah. So let's take the the word machine part first. Predicting words that humans part first. Predicting words that humans part first. Predicting words that humans wrote often requires solving a harder wrote often requires solving a harder wrote often requires solving a harder problem than the human who wrote them. problem than the human who wrote them. problem than the human who wrote them. So suppose that you go and inject a drug So suppose that you go and inject a drug So suppose that you go and inject a drug in a rat and you're like, you know, it's in a rat and you're like, you know, it's in a rat and you're like, you know, it's like you write down the chemical nature like you write down the chemical nature like you write down the chemical nature of the drug. You inject it into the rat. of the drug. You inject it into the rat. of the drug. You inject it into the rat. You see that the rat dies and so you're You see that the rat dies and so you're You see that the rat dies and so you're like, when I put that drug into the rat, like, when I put that drug into the rat, like, when I put that drug into the rat, the rat died. Now suppose you're the rat died. Now suppose you're the rat died. Now suppose you're training an AI and the AI sees the training an AI and the AI sees the training an AI and the AI sees the chemical nature of the drug. It sees chemical nature of the drug. It sees chemical nature of the drug. It sees when I put that drug into the rat, the when I put that drug into the rat, the when I put that drug into the rat, the rat blank.

rat blank. rat blank. The human who wrote it down gets to just The human who wrote it down gets to just The human who wrote it down gets to just look at what happened to the rat. look at what happened to the rat. look at what happened to the rat. The AI predicting what was written does The AI predicting what was written does The AI predicting what was written does not get to just look at the rat. So not get to just look at the rat. So not get to just look at the rat. So training AIs to predict human text is training AIs to predict human text is training AIs to predict human text is training them to be potentially smarter training them to be potentially smarter training them to be potentially smarter than the humans than the humans than the humans because they need to be able to answer because they need to be able to answer because they need to be able to answer these qu they need to be able to predict these qu they need to be able to predict these qu they need to be able to predict they need to be able to like uh fill in they need to be able to like uh fill in they need to be able to like uh fill in the blanks where humans were just the blanks where humans were just the blanks where humans were just writing down what they saw and there's writing down what they saw and there's writing down what they saw and there's just so I understand technologically just so I understand technologically just so I understand technologically there is no knowledge they have though there is no knowledge they have though there is no knowledge they have though each time and there are there are ways each time and there are there are ways each time and there are there are ways of kind of mitigating these each time it of kind of mitigating these each time it of kind of mitigating these each time it is effectively rereading but because of is effectively rereading but because of is effectively rereading but because of training it gets more accurate at training it gets more accurate at training it gets more accurate at certain things. Uh I mean somehow as you certain things. Uh I mean somehow as you certain things. Uh I mean somehow as you tune the knobs somehow it's getting tune the knobs somehow it's getting tune the knobs somehow it's getting information in there and we don't know information in there and we don't know information in there and we don't know how. how. how. >> So it's much easier than that. We're >> So it's much easier than that. We're >> So it's much easier than that. We're humans. We have a brain. Brains are made humans. We have a brain. Brains are made humans. We have a brain. Brains are made of neurons. Then we try to copy that on of neurons. Then we try to copy that on of neurons. Then we try to copy that on a computer. We simplify it but we create a computer. We simplify it but we create a computer. We simplify it but we create a neural network. So we're making a neural network. So we're making a neural network. So we're making artificial brains just like with human artificial brains just like with human artificial brains just like with human brains. With cognitive science, we don't brains. With cognitive science, we don't brains. With cognitive science, we don't really understand how you function, how really understand how you function, how really understand how you function, how you learn, where in your brain certain you learn, where in your brain certain you learn, where in your brain certain memories are stored. We have some memories are stored. We have some memories are stored. We have some glimpses of understanding this neuron glimpses of understanding this neuron glimpses of understanding this neuron fires then you see a face but there is fires then you see a face but there is fires then you see a face but there is no complete picture and so a lot of no complete picture and so a lot of no complete picture and so a lot of times you can't get intuitive times you can't get intuitive times you can't get intuitive understanding of what's going on then understanding of what's going on then understanding of what's going on then you just think about it as artificial you just think about it as artificial you just think about it as artificial persons. It's not exact mapping but it persons. It's not exact mapping but it persons. It's not exact mapping but it helps. So if you send a child through 12 helps. So if you send a child through 12 helps. So if you send a child through 12 years of education they get lots of years of education they get lots of years of education they get lots of problems to look at and then they problems to look at and then they problems to look at and then they graduate and become a little better at graduate and become a little better at graduate and become a little better at solving problems. This is what we're

solving problems. This is what we're solving problems. This is what we're trying to replicate here. People trying to replicate here. People trying to replicate here. People complain that it takes a lot of money to complain that it takes a lot of money to complain that it takes a lot of money to train those very, you know, intense train those very, you know, intense train those very, you know, intense process. You forget that it takes 20 process. You forget that it takes 20 process. You forget that it takes 20 years to train a human and they are not years to train a human and they are not years to train a human and they are not general super intelligences. They are general super intelligences. They are general super intelligences. They are very narrow. We're lucky if they very narrow. We're lucky if they very narrow. We're lucky if they graduate with a bachelors. So a lot of graduate with a bachelors. So a lot of graduate with a bachelors. So a lot of it is exactly the same. Can we make safe it is exactly the same. Can we make safe it is exactly the same. Can we make safe humans for example? We invented humans for example? We invented humans for example? We invented religion, ethics, lie detector tests and religion, ethics, lie detector tests and religion, ethics, lie detector tests and yet human safety is still unsolved yet human safety is still unsolved yet human safety is still unsolved problem. Now you have something more problem. Now you have something more problem. Now you have something more alien. doesn't have physical body, alien. doesn't have physical body, alien. doesn't have physical body, doesn't have biological needs. So there doesn't have biological needs. So there doesn't have biological needs. So there are additional complications. But all are additional complications. But all are additional complications. But all the problems we face with humans still the problems we face with humans still the problems we face with humans still there, safety problems, crime, all that there, safety problems, crime, all that there, safety problems, crime, all that stays and problems with understanding stays and problems with understanding stays and problems with understanding what motivates a human to do something. what motivates a human to do something. what motivates a human to do something. Why do we get mental disorders? All that Why do we get mental disorders? All that Why do we get mental disorders? All that shows up there. shows up there. shows up there. >> And we still don't if someone is a >> And we still don't if someone is a >> And we still don't if someone is a serial killer and we look at their serial killer and we look at their serial killer and we look at their brain, we can't often figure out exactly brain, we can't often figure out exactly brain, we can't often figure out exactly why why they made the decision to kill a why why they made the decision to kill a why why they made the decision to kill a bunch of bunch of bunch of >> and you can't be like, "Oh, I'll go >> and you can't be like, "Oh, I'll go >> and you can't be like, "Oh, I'll go change these neurons so that they stop change these neurons so that they stop change these neurons so that they stop being a serial killer." we just like being a serial killer." we just like being a serial killer." we just like don't have that capacity with the AI.

don't have that capacity with the AI. don't have that capacity with the AI. >> This is one of the big questions that >> This is one of the big questions that >> This is one of the big questions that people want to know is people want to know is people want to know is there's this sort of illusion of control there's this sort of illusion of control there's this sort of illusion of control with AI. Um if we don't even fully with AI. Um if we don't even fully with AI. Um if we don't even fully understand how modern neural networks understand how modern neural networks understand how modern neural networks think, why do companies believe they can think, why do companies believe they can think, why do companies believe they can control any form of super intelligence? control any form of super intelligence? control any form of super intelligence? If we don't understand how they think, If we don't understand how they think, If we don't understand how they think, >> it it's worse if they understood how the >> it it's worse if they understood how the >> it it's worse if they understood how the system works. Then recursive system works. Then recursive system works. Then recursive self-improvement becomes much easier. self-improvement becomes much easier. self-improvement becomes much easier. You get faster takeoff. Right now the You get faster takeoff. Right now the You get faster takeoff. Right now the model doesn't understand its own. model doesn't understand its own. model doesn't understand its own. thinking. thinking. thinking. >> Do we understand how these systems >> Do we understand how these systems >> Do we understand how these systems think, Andy? think, Andy? think, Andy? >> I mean, I agree. These are black boxes >> I mean, I agree. These are black boxes >> I mean, I agree. These are black boxes in some pretty important ways. I'm just in some pretty important ways. I'm just in some pretty important ways. I'm just less terrified by that than a lot of less terrified by that than a lot of less terrified by that than a lot of other people are. There are lots of other people are. There are lots of other people are. There are lots of things we don't understand very well. things we don't understand very well. things we don't understand very well. Can we contain things that we don't Can we contain things that we don't Can we contain things that we don't understand perfectly? Yes, we can. I understand perfectly? Yes, we can. I understand perfectly? Yes, we can. I think Open AI did a we've talked about think Open AI did a we've talked about think Open AI did a we've talked about it did a lousy job of building the it did a lousy job of building the it did a lousy job of building the containment for the uh AI that they that containment for the uh AI that they that containment for the uh AI that they that they stood up to try to exploit to try they stood up to try to exploit to try they stood up to try to exploit to try to crack security problems that went out to crack security problems that went out to crack security problems that went out into the outside world. They did a lousy into the outside world. They did a lousy into the outside world. They did a lousy job of building the virtual sandbox that job of building the virtual sandbox that job of building the virtual sandbox that it was where it was supposed to have to it was where it was supposed to have to it was where it was supposed to have to re where supposed to remain and it re where supposed to remain and it re where supposed to remain and it didn't remain. That doesn't mean that didn't remain. That doesn't mean that didn't remain. That doesn't mean that it's impossible. It means OpenAI did a it's impossible. It means OpenAI did a it's impossible. It means OpenAI did a pretty bad job of And is that a function pretty bad job of And is that a function pretty bad job of And is that a function of those humans and their intelligence?

of those humans and their intelligence? of those humans and their intelligence? >> I think it's just a function of pretty >> I think it's just a function of pretty >> I think it's just a function of pretty lousy security protocol lousy security protocol lousy security protocol >> based by from human intelligence. The >> based by from human intelligence. The >> based by from human intelligence. The idea that sandbox was built by human idea that sandbox was built by human idea that sandbox was built by human intelligence. It sounds like there was a intelligence. It sounds like there was a intelligence. It sounds like there was a deficit in human intelligence deficit in human intelligence deficit in human intelligence potentially. potentially. potentially. >> Sure. But there are, you know, people >> Sure. But there are, you know, people >> Sure. But there are, you know, people who drive cars in telephone calls. Does who drive cars in telephone calls. Does who drive cars in telephone calls. Does that mean we can't drive? Shouldn't make that mean we can't drive? Shouldn't make that mean we can't drive? Shouldn't make them super intelligent. them super intelligent. them super intelligent. >> No, but you wouldn't I mean arguably >> No, but you wouldn't I mean arguably >> No, but you wouldn't I mean arguably >> like this is what we're trying to solve >> like this is what we're trying to solve >> like this is what we're trying to solve for at the moment. No, the fact is a for at the moment. No, the fact is a for at the moment. No, the fact is a mist like it feels I don't know the mist like it feels I don't know the mist like it feels I don't know the details. It feels to me like they made details. It feels to me like they made details. It feels to me like they made some fairly basic mistakes in setting up some fairly basic mistakes in setting up some fairly basic mistakes in setting up this confined environment. I this confined environment. I this confined environment. I >> I think that wasn't true in the open >> I think that wasn't true in the open >> I think that wasn't true in the open case. It was true in a lot of the cases case. It was true in a lot of the cases case. It was true in a lot of the cases but not the open. but not the open. but not the open. >> That doesn't mean >> That doesn't mean >> That doesn't mean >> that we are unable to control this black >> that we are unable to control this black >> that we are unable to control this black box. That does not necessarily follow. box. That does not necessarily follow. box. That does not necessarily follow. >> I get that. It's just at a time when >> I get that. It's just at a time when >> I get that. It's just at a time when that the um you got a human trying to that the um you got a human trying to that the um you got a human trying to contain something that is smarter than contain something that is smarter than contain something that is smarter than it. One would con logically conclude it. One would con logically conclude it. One would con logically conclude that if the thing is smarter than I am that if the thing is smarter than I am that if the thing is smarter than I am and I'm trying to contain it, it would and I'm trying to contain it, it would and I'm trying to contain it, it would be better at knowing the exploits or be better at knowing the exploits or be better at knowing the exploits or vulnerabilities. In my own um vulnerabilities. In my own um vulnerabilities. In my own um >> saying if you put Einstein in a jail, >> saying if you put Einstein in a jail, >> saying if you put Einstein in a jail, you could never contain him. I don't you could never contain him. I don't you could never contain him. I don't agree with that.

agree with that. agree with that. >> Put him in jail with an internet >> Put him in jail with an internet >> Put him in jail with an internet connection and use a digital mind connection and use a digital mind connection and use a digital mind question. question. question. >> Yeah. Yeah. That that's probably an >> Yeah. Yeah. That that's probably an >> Yeah. Yeah. That that's probably an squar keep Einstein in prison. That's squar keep Einstein in prison. That's squar keep Einstein in prison. That's the question. the question. the question. >> The hacking accident, as far as I know, >> The hacking accident, as far as I know, >> The hacking accident, as far as I know, they found zero day exploits, which they found zero day exploits, which they found zero day exploits, which means completely novel exploits. no means completely novel exploits. no means completely novel exploits. no human knew about. It wasn't just poor human knew about. It wasn't just poor human knew about. It wasn't just poor setup. The password is, you know, quy. setup. The password is, you know, quy. setup. The password is, you know, quy. It was a brand new escape It was a brand new escape It was a brand new escape >> for multiple zero days. So, a zero day >> for multiple zero days. So, a zero day >> for multiple zero days. So, a zero day attack is an attack that the defenders attack is an attack that the defenders attack is an attack that the defenders have had zero days to handle. It's cyber have had zero days to handle. It's cyber have had zero days to handle. It's cyber security lingo. Um, and so when we say security lingo. Um, and so when we say security lingo. Um, and so when we say that they use zero day attacks, what we that they use zero day attacks, what we that they use zero day attacks, what we mean is that these AIs were finding bugs mean is that these AIs were finding bugs mean is that these AIs were finding bugs in the software that the humans had no in the software that the humans had no in the software that the humans had no knowledge of and they were finding knowledge of and they were finding knowledge of and they were finding multiple of these bugs. One of these multiple of these bugs. One of these multiple of these bugs. One of these bugs usually doesn't let you break out. bugs usually doesn't let you break out. bugs usually doesn't let you break out. It's sort of like if you find a crack in It's sort of like if you find a crack in It's sort of like if you find a crack in the wall over here and you find a crack the wall over here and you find a crack the wall over here and you find a crack on the outside of the wall over there, on the outside of the wall over there, on the outside of the wall over there, then you just need to like dig a little then you just need to like dig a little then you just need to like dig a little bit to connect those cracks. bit to connect those cracks. bit to connect those cracks. >> You don't sell those for millions of >> You don't sell those for millions of >> You don't sell those for millions of dollars on the dark market if you find dollars on the dark market if you find dollars on the dark market if you find one. So, difficult to find one. So, difficult to find one. So, difficult to find >> in how just so I understand for the >> in how just so I understand for the >> in how just so I understand for the listeners swap. listeners swap. listeners swap. >> Is this is a zero day always a novel way >> Is this is a zero day always a novel way >> Is this is a zero day always a novel way that no one has ever used to break that no one has ever used to break that no one has ever used to break anything before or is it just for the anything before or is it just for the anything before or is it just for the unique situation like so was it a zero unique situation like so was it a zero unique situation like so was it a zero day for a thing in hugging face versus a day for a thing in hugging face versus a day for a thing in hugging face versus a novel new way of hacking in general? Um, novel new way of hacking in general? Um, novel new way of hacking in general? Um, so it was uh they weren't like totally so it was uh they weren't like totally so it was uh they weren't like totally novel hacking techniques.

novel hacking techniques. novel hacking techniques. >> That's kind of why I was g not to say >> That's kind of why I was g not to say >> That's kind of why I was g not to say it's not bad, but just like there's a it's not bad, but just like there's a it's not bad, but just like there's a difference between it came up with a difference between it came up with a difference between it came up with a brand new way to do something. brand new way to do something. brand new way to do something. >> Actually, I'm not sure we have all of >> Actually, I'm not sure we have all of >> Actually, I'm not sure we have all of the vulnerabilities released, but mostly the vulnerabilities released, but mostly the vulnerabilities released, but mostly it was like it so it was indeed sort of it was like it so it was indeed sort of it was like it so it was indeed sort of like finding ways that humans tend to like finding ways that humans tend to like finding ways that humans tend to make mistakes make mistakes make mistakes >> and finding another one of those in a >> and finding another one of those in a >> and finding another one of those in a place they hadn't seen. But this is place they hadn't seen. But this is place they hadn't seen. But this is actually such a hard task that as Roman actually such a hard task that as Roman actually such a hard task that as Roman says, humans can be paid $100,000 to $5 says, humans can be paid $100,000 to $5 says, humans can be paid $100,000 to $5 million as a bounty for this type of million as a bounty for this type of million as a bounty for this type of exploit. So the amount of labor it takes exploit. So the amount of labor it takes exploit. So the amount of labor it takes to find these for a human is actually to find these for a human is actually to find these for a human is actually pretty high. pretty high. pretty high. >> Let me just explain that cuz most people >> Let me just explain that cuz most people >> Let me just explain that cuz most people don't know what a bounty is in this don't know what a bounty is in this don't know what a bounty is in this regard. regard. regard. >> So there are certain types of bugs where >> So there are certain types of bugs where >> So there are certain types of bugs where if you find a bug in software that lets if you find a bug in software that lets if you find a bug in software that lets you take control of someone's computer, you take control of someone's computer, you take control of someone's computer, one thing you can do is you can use it one thing you can do is you can use it one thing you can do is you can use it to take over a lot of computers. Another to take over a lot of computers. Another to take over a lot of computers. Another thing you can do is you can go to the thing you can do is you can go to the thing you can do is you can go to the people with that software and say your people with that software and say your people with that software and say your software is broken. Do you want me to software is broken. Do you want me to software is broken. Do you want me to tell you where the bug is? I can show tell you where the bug is? I can show tell you where the bug is? I can show you that I can take your stuff over. And you that I can take your stuff over. And you that I can take your stuff over. And so that people will sort of report the so that people will sort of report the so that people will sort of report the bugs. Uh people uh will often offer bugs. Uh people uh will often offer bugs. Uh people uh will often offer money to the good guys and then you know money to the good guys and then you know money to the good guys and then you know the bad guys will often also offer money the bad guys will often also offer money the bad guys will often also offer money sometimes try to outbid them and so you sometimes try to outbid them and so you sometimes try to outbid them and so you can make somewhere between hundreds of can make somewhere between hundreds of can make somewhere between hundreds of thousands and millions of dollars if you thousands and millions of dollars if you thousands and millions of dollars if you personally can find these issues. I personally can find these issues. I personally can find these issues. I think there's a rare point of agreement think there's a rare point of agreement think there's a rare point of agreement across the four of us here, which is across the four of us here, which is across the four of us here, which is that we are in a new era of cyber that we are in a new era of cyber that we are in a new era of cyber security as of this explain. We we are security as of this explain. We we are security as of this explain. We we are in very new territory for reasons that in very new territory for reasons that in very new territory for reasons that we've talked about. We've got these we've talked about. We've got these we've talked about. We've got these large numbers of agents who are grinding large numbers of agents who are grinding large numbers of agents who are grinding away and they carry around or they had

away and they carry around or they had away and they carry around or they had access to a huge number of keys to go access to a huge number of keys to go access to a huge number of keys to go open all the different locks that they open all the different locks that they open all the different locks that they faced and they did this bizarly good job faced and they did this bizarly good job faced and they did this bizarly good job of it and got a long way. I think that's of it and got a long way. I think that's of it and got a long way. I think that's absolutely true. I think all four of us absolutely true. I think all four of us absolutely true. I think all four of us are are in rare alignment on that at are are in rare alignment on that at are are in rare alignment on that at this table. [gasps] this table. [gasps] this table. [gasps] >> If you are and given that we're in this >> If you are and given that we're in this >> If you are and given that we're in this era, do you know what you really really era, do you know what you really really era, do you know what you really really really want on your side? really want on your side? really want on your side? >> I know what you're going to say. >> I know what you're going to say. >> I know what you're going to say. >> Tell me. >> Tell me. >> Tell me. >> AI. >> AI. >> AI. >> Really, really good AI. Does anybody >> Really, really good AI. Does anybody >> Really, really good AI. Does anybody disagree with that? Do you want do you disagree with that? Do you want do you disagree with that? Do you want do you want to give up leadership on AI in this want to give up leadership on AI in this want to give up leadership on AI in this era of cyber security? era of cyber security? era of cyber security? >> It's a good point because China are >> It's a good point because China are >> It's a good point because China are going to have a great weapon. Uh my going to have a great weapon. Uh my going to have a great weapon. Uh my stance is pretty neutral on what to do stance is pretty neutral on what to do stance is pretty neutral on what to do about the hacking AIs and the coming about the hacking AIs and the coming about the hacking AIs and the coming cyber apocalypse are pretty neutral cyber apocalypse are pretty neutral cyber apocalypse are pretty neutral about what to do about you know whether about what to do about you know whether about what to do about you know whether we should put the AIs in uh the drones we should put the AIs in uh the drones we should put the AIs in uh the drones and save human lives or whether we and save human lives or whether we and save human lives or whether we should avoid that because then what if should avoid that because then what if should avoid that because then what if the drones blah blah blah. the drones blah blah blah. the drones blah blah blah. >> This is a graph showing China versus the >> This is a graph showing China versus the >> This is a graph showing China versus the United States. You don't really need to United States. You don't really need to United States. You don't really need to see the detail. You can see the outline see the detail. You can see the outline see the detail. You can see the outline of the graph. of the graph. of the graph. >> Are you neutral in falling behind our >> Are you neutral in falling behind our >> Are you neutral in falling behind our adversaries in AI? adversaries in AI? adversaries in AI? >> I think that if anyone builds a rogue >> I think that if anyone builds a rogue >> I think that if anyone builds a rogue super intelligence, everybody dies.

super intelligence, everybody dies. super intelligence, everybody dies. That's not an answer in my question. That's not an answer in my question. That's not an answer in my question. >> I mean, what part of AI are you asking >> I mean, what part of AI are you asking >> I mean, what part of AI are you asking whether we should fall behind on? Like I whether we should fall behind on? Like I whether we should fall behind on? Like I I don't think we should fall behind on I don't think we should fall behind on I don't think we should fall behind on cyber hacking. I do think that we should cyber hacking. I do think that we should cyber hacking. I do think that we should not be racing to destroy the world with not be racing to destroy the world with not be racing to destroy the world with American hands instead of Chinese ones American hands instead of Chinese ones American hands instead of Chinese ones because we really want to be killed by, because we really want to be killed by, because we really want to be killed by, you know, we we care whether the killer you know, we we care whether the killer you know, we we care whether the killer robots talk English or Mandarin, if robots talk English or Mandarin, if robots talk English or Mandarin, if that's what you're asking. that's what you're asking. that's what you're asking. >> I find it interesting. I find that >> I find it interesting. I find that >> I find it interesting. I find that you're dodging these questions or you're you're dodging these questions or you're you're dodging these questions or you're neutral on them because they're neutral on them because they're neutral on them because they're inconvenient for your argument that we inconvenient for your argument that we inconvenient for your argument that we need to be calling a halt to this. I'm need to be calling a halt to this. I'm need to be calling a halt to this. I'm neutral. Let me finish please. There neutral. Let me finish please. There neutral. Let me finish please. There will be risks and harms to all kinds of will be risks and harms to all kinds of will be risks and harms to all kinds of things if the United States calls a halt things if the United States calls a halt things if the United States calls a halt to AI. And maybe you're indifferent if to AI. And maybe you're indifferent if to AI. And maybe you're indifferent if the Chinese get ahead of us and then the Chinese get ahead of us and then the Chinese get ahead of us and then they make super intelligence and it and they make super intelligence and it and they make super intelligence and it and and it kills us all. Are you or that's a and it kills us all. Are you or that's a and it kills us all. Are you or that's a >> I do not think we should do a domestic >> I do not think we should do a domestic >> I do not think we should do a domestic pause. pause. pause. >> Do you think there's any hope for a >> Do you think there's any hope for a >> Do you think there's any hope for a global pause? global pause? global pause? >> Absolutely. Do you think the Chinese and >> Absolutely. Do you think the Chinese and >> Absolutely. Do you think the Chinese and our and the Iranians and the North our and the Iranians and the North our and the Iranians and the North Koreans and the Russians are a going to Koreans and the Russians are a going to Koreans and the Russians are a going to come to a table with us, hammer out an come to a table with us, hammer out an come to a table with us, hammer out an agreement, and b abide by it when agreement, and b abide by it when agreement, and b abide by it when verifiability is really low.

verifiability is really low. verifiability is really low. Verifiability doesn't need to be really Verifiability doesn't need to be really Verifiability doesn't need to be really low, low, low, >> gentlemen. That is shockingly naive. >> gentlemen. That is shockingly naive. >> gentlemen. That is shockingly naive. >> Training a super shockingly naive. >> Training a super shockingly naive. >> Training a super shockingly naive. >> Training one of these AIs, training one >> Training one of these AIs, training one >> Training one of these AIs, training one of these frontier AIs takes a 100,000 of of these frontier AIs takes a 100,000 of of these frontier AIs takes a 100,000 of the most advanced computer chip humanity the most advanced computer chip humanity the most advanced computer chip humanity can produce. This is practically the can produce. This is practically the can produce. This is practically the peak output of the global supply chain. peak output of the global supply chain. peak output of the global supply chain. Many parts of that supply chain are Many parts of that supply chain are Many parts of that supply chain are controlled by the US and US allies. controlled by the US and US allies. controlled by the US and US allies. There's roughly one fab in Taiwan that There's roughly one fab in Taiwan that There's roughly one fab in Taiwan that can produce these trips. There's roughly can produce these trips. There's roughly can produce these trips. There's roughly one country in the world that can one country in the world that can one country in the world that can produce the lithography machines that produce the lithography machines that produce the lithography machines that are critical in the process, which is are critical in the process, which is are critical in the process, which is the Netherlands, which is an ally. To the Netherlands, which is an ally. To the Netherlands, which is an ally. To assemble a 100,000 of these trips to do assemble a 100,000 of these trips to do assemble a 100,000 of these trips to do one of these training runs that can make one of these training runs that can make one of these training runs that can make the more dangerous type of AI, you need the more dangerous type of AI, you need the more dangerous type of AI, you need to assemble them into an enormous data to assemble them into an enormous data to assemble them into an enormous data center that costs tons of money that center that costs tons of money that center that costs tons of money that draws down electricity comparable to a draws down electricity comparable to a draws down electricity comparable to a city and run it for the better part of a city and run it for the better part of a city and run it for the better part of a year. You can see that infrastructure year. You can see that infrastructure year. You can see that infrastructure from space. from space. from space. China has much less trip capacity than China has much less trip capacity than China has much less trip capacity than the US does. It is absolutely possible the US does. It is absolutely possible the US does. It is absolutely possible if we were trying for the US to say we if we were trying for the US to say we if we were trying for the US to say we are going to monitor where these chips are going to monitor where these chips are going to monitor where these chips go. We are going to monitor heavy go. We are going to monitor heavy go. We are going to monitor heavy concentrations of these. These are not concentrations of these. These are not concentrations of these. These are not consumer amounts of chips. These are consumer amounts of chips. These are consumer amounts of chips. These are huge amounts of chips. And to say we are huge amounts of chips. And to say we are huge amounts of chips. And to say we are going to make sure that there is no going to make sure that there is no going to make sure that there is no training run trying to make a super training run trying to make a super training run trying to make a super intelligence in here. You can mess intelligence in here. You can mess intelligence in here. You can mess around with the cyber stuff whatever you around with the cyber stuff whatever you around with the cyber stuff whatever you want because that does not end humanity.

want because that does not end humanity. want because that does not end humanity. I am concerned with the stuff that can I am concerned with the stuff that can I am concerned with the stuff that can end humanity. The reason I'm being end humanity. The reason I'm being end humanity. The reason I'm being neutral on your questions is because neutral on your questions is because neutral on your questions is because humanity is going to die if we do not humanity is going to die if we do not humanity is going to die if we do not stop creating super intelligence. And we stop creating super intelligence. And we stop creating super intelligence. And we could absolutely could absolutely could absolutely track where those trips are going and track where those trips are going and track where those trips are going and stop them from doing these training runs stop them from doing these training runs stop them from doing these training runs while allowing them to do economically while allowing them to do economically while allowing them to do economically productive stuff that we already know is productive stuff that we already know is productive stuff that we already know is safe. And it would be far easier than safe. And it would be far easier than safe. And it would be far easier than uranium, which is a rock you dig out of uranium, which is a rock you dig out of uranium, which is a rock you dig out of the ground and spin around really fast. the ground and spin around really fast. the ground and spin around really fast. How do you discern between a training How do you discern between a training How do you discern between a training run for super intelligence and the run for super intelligence and the run for super intelligence and the training run for cyber security? Because training run for cyber security? Because training run for cyber security? Because you're referring, I assume, to the you're referring, I assume, to the you're referring, I assume, to the 100,000 chips that are in Stargate 100,000 chips that are in Stargate 100,000 chips that are in Stargate Abene, right? the ones that we used to Abene, right? the ones that we used to Abene, right? the ones that we used to train Astra because how would you train Astra because how would you train Astra because how would you discern between training for super discern between training for super discern between training for super intelligence in Abalene which does not intelligence in Abalene which does not intelligence in Abalene which does not have as many chips as they say but have as many chips as they say but have as many chips as they say but nevertheless and how like a super nevertheless and how like a super nevertheless and how like a super intelligence because I I actually have intelligence because I I actually have intelligence because I I actually have my own feelings here but just I'm not my own feelings here but just I'm not my own feelings here but just I'm not sure how you square the circle of how do sure how you square the circle of how do sure how you square the circle of how do you stop China even though China is you stop China even though China is you stop China even though China is getting their LM based on distilling getting their LM based on distilling getting their LM based on distilling arts we know that arts we know that arts we know that >> but but the thing is it's like how do >> but but the thing is it's like how do >> but but the thing is it's like how do you discern because you can't really you discern because you can't really you discern because you can't really >> you play it safe right now the way we >> you play it safe right now the way we >> you play it safe right now the way we make these things smarter is to make make these things smarter is to make make these things smarter is to make them far larger.

them far larger. them far larger. >> Yes. >> Yes. >> Yes. >> So what you do is you say, "Hey, look, >> So what you do is you say, "Hey, look, >> So what you do is you say, "Hey, look, training runs of this size that risks training runs of this size that risks training runs of this size that risks destroying everybody. No one's going to destroying everybody. No one's going to destroying everybody. No one's going to do it." do it." do it." >> This point about can we get China to >> This point about can we get China to >> This point about can we get China to cooperate and can we check that they are cooperate and can we check that they are cooperate and can we check that they are >> fundamentally we should so a >> fundamentally we should so a >> fundamentally we should so a fundamentally we should be trying to get fundamentally we should be trying to get fundamentally we should be trying to get them to cooperate. them to cooperate. them to cooperate. >> Yeah, >> Yeah, >> Yeah, >> it is personal self-interest. >> it is personal self-interest. >> it is personal self-interest. Nobody wins if they get destroyed. You Nobody wins if they get destroyed. You Nobody wins if they get destroyed. You don't make money. You don't stay in don't make money. You don't stay in don't make money. You don't stay in power. Communist Party of China is power. Communist Party of China is power. Communist Party of China is really good at staying in power. really good at staying in power. really good at staying in power. President Trump is also excellent. President Trump is also excellent. President Trump is also excellent. >> And you think they're going to sign and >> And you think they're going to sign and >> And you think they're going to sign and abide by an agreement that leaves them abide by an agreement that leaves them abide by an agreement that leaves them permanently in secondly permanently in secondly permanently in secondly in second place? in second place? in second place? >> No. No one is permanently in second >> No. No one is permanently in second >> No. No one is permanently in second place if nobody is building the rogue place if nobody is building the rogue place if nobody is building the rogue super intelligence. super intelligence. super intelligence. >> They have a government one trick ponies, >> They have a government one trick ponies, >> They have a government one trick ponies, man. It's like what you're fixated on man. It's like what you're fixated on man. It's like what you're fixated on this one thing and nothing else matters this one thing and nothing else matters this one thing and nothing else matters to you. to you. to you. >> You got it now. That nothing else other >> You got it now. That nothing else other >> You got it now. That nothing else other than saving humanity. Everything is than saving humanity. Everything is than saving humanity. Everything is secondary. Absolutely. China is our secondary. Absolutely. China is our secondary. Absolutely. China is our biggest trading partner. Everything we biggest trading partner. Everything we biggest trading partner. Everything we have is made in China. They have not have is made in China. They have not have is made in China. They have not attacked us. They haven't. If you look attacked us. They haven't. If you look attacked us. They haven't. If you look at the last 30 years, how many wars did at the last 30 years, how many wars did at the last 30 years, how many wars did they start? Not so bad. We can make a they start? Not so bad. We can make a they start? Not so bad. We can make a deal. And they have government of deal. And they have government of deal. And they have government of engineers and scientists, not lawyers.

engineers and scientists, not lawyers. engineers and scientists, not lawyers. They understand scientific arguments. They understand scientific arguments. They understand scientific arguments. There are panels, workshops. American There are panels, workshops. American There are panels, workshops. American computer scientists, Chinese get computer scientists, Chinese get computer scientists, Chinese get together. That means communist party together. That means communist party together. That means communist party authorized those meetings. They are authorized those meetings. They are authorized those meetings. They are talking about it. And there is a lot of talking about it. And there is a lot of talking about it. And there is a lot of consensus on this technology. consensus on this technology. consensus on this technology. >> And you can build things into these >> And you can build things into these >> And you can build things into these computer chips to make this stuff more computer chips to make this stuff more computer chips to make this stuff more verifiable. You can build location verifiable. You can build location verifiable. You can build location tracking devices into these. tracking devices into these. tracking devices into these. >> So, so this technology is controllable. >> So, so this technology is controllable. >> So, so this technology is controllable. >> Absolutely. The super intelligence is >> Absolutely. The super intelligence is >> Absolutely. The super intelligence is not controllable. not controllable. not controllable. >> There's a separation between software >> There's a separation between software >> There's a separation between software and hardware which you did. and hardware which you did. and hardware which you did. >> I am not saying we are going to die. I >> I am not saying we are going to die. I >> I am not saying we are going to die. I am saying that we need to actually not am saying that we need to actually not am saying that we need to actually not build the rogue super intelligences. build the rogue super intelligences. build the rogue super intelligences. Humanity absolutely could say we are Humanity absolutely could say we are Humanity absolutely could say we are going to track where the chips go. going to track where the chips go. going to track where the chips go. The US absolutely could say that we fear The US absolutely could say that we fear The US absolutely could say that we fear for our lives if China starts a super for our lives if China starts a super for our lives if China starts a super intelligence training run and make it intelligence training run and make it intelligence training run and make it very diplomatically clear to China that very diplomatically clear to China that very diplomatically clear to China that we think this would kill you and us and we think this would kill you and us and we think this would kill you and us and there's no benefit and we are not going there's no benefit and we are not going there's no benefit and we are not going to do it because we think it would kill to do it because we think it would kill to do it because we think it would kill you and us and there's no benefit and we you and us and there's no benefit and we you and us and there's no benefit and we think you should sign this nice here think you should sign this nice here think you should sign this nice here treaty because we think it would kill treaty because we think it would kill treaty because we think it would kill all of us and there'd be no benefit. But all of us and there'd be no benefit. But all of us and there'd be no benefit. But if you don't we're going to fear for our if you don't we're going to fear for our if you don't we're going to fear for our lives and you know treat that lives and you know treat that lives and you know treat that as we would to defend ourselves. We as we would to defend ourselves. We as we would to defend ourselves. We should separate the question of can we should separate the question of can we should separate the question of can we put a stop to it.

put a stop to it. put a stop to it. >> Uhhuh. >> Uhhuh. >> Uhhuh. >> Is it possible if world governments >> Is it possible if world governments >> Is it possible if world governments realized just how crazy this stuff is? realized just how crazy this stuff is? realized just how crazy this stuff is? Could they put a stop to it? Could it be Could they put a stop to it? Could it be Could they put a stop to it? Could it be monitored? Could it be verified? Could monitored? Could it be verified? Could monitored? Could it be verified? Could it be enforced? That's one question. it be enforced? That's one question. it be enforced? That's one question. There's a separate question which is There's a separate question which is There's a separate question which is will people realize? will people realize? will people realize? >> If it got cheaper to train super >> If it got cheaper to train super >> If it got cheaper to train super intelligence, intelligence, intelligence, >> then we'd be in a bad spot. >> then we'd be in a bad spot. >> then we'd be in a bad spot. >> Your approach would no longer be >> Your approach would no longer be >> Your approach would no longer be effective. effective. effective. >> That's right. >> That's right. >> That's right. >> Because more countries could capitalize >> Because more countries could capitalize >> Because more countries could capitalize on the opportunity. on the opportunity. on the opportunity. >> That's right. But we're not there yet. >> That's right. But we're not there yet. >> That's right. But we're not there yet. So, how do you rebut that point? So, how do you rebut that point? So, how do you rebut that point? >> Yeah. So, I would say it looks to me >> Yeah. So, I would say it looks to me >> Yeah. So, I would say it looks to me like there is a danger of the the future like there is a danger of the the future like there is a danger of the the future training runs getting there and that is training runs getting there and that is training runs getting there and that is enough to stop doing it when humanity is enough to stop doing it when humanity is enough to stop doing it when humanity is at risk. at risk. at risk. >> Sure. >> Sure. >> Sure. >> Uh I think that you also need to have an >> Uh I think that you also need to have an >> Uh I think that you also need to have an answer about what happens if it gets answer about what happens if it gets answer about what happens if it gets much much cheaper to do this stuff. I much much cheaper to do this stuff. I much much cheaper to do this stuff. I think it's a hard problem. I would think it's a hard problem. I would think it's a hard problem. I would recommend that we also put a taboo on recommend that we also put a taboo on recommend that we also put a taboo on research of trying to make AI super research of trying to make AI super research of trying to make AI super cheap to train if it would lead in the cheap to train if it would lead in the cheap to train if it would lead in the direction of super intelligence. Just direction of super intelligence. Just direction of super intelligence. Just like we have a research taboo on making like we have a research taboo on making like we have a research taboo on making your own nuclear weapons or finding out your own nuclear weapons or finding out your own nuclear weapons or finding out how to make like let civilians make how to make like let civilians make how to make like let civilians make nuclear weapons. I would say trying to nuclear weapons. I would say trying to nuclear weapons. I would say trying to find ways to let civilians train super find ways to let civilians train super find ways to let civilians train super intelligences should be treated the same intelligences should be treated the same intelligences should be treated the same as trying to find ways to like let as trying to find ways to like let as trying to find ways to like let civilians propagate nukes. We're sort of civilians propagate nukes. We're sort of civilians propagate nukes. We're sort of like don't do that research in the like don't do that research in the like don't do that research in the public sphere. that fi that seems um public sphere. that fi that seems um public sphere. that fi that seems um like wishful thinking in the context like wishful thinking in the context like wishful thinking in the context that these will become public companies that these will become public companies that these will become public companies who are incentivized to bring down who are incentivized to bring down who are incentivized to bring down costs.

costs. costs. >> It's a it's a tough position. I think >> It's a it's a tough position. I think >> It's a it's a tough position. I think right now the thing that brings down right now the thing that brings down right now the thing that brings down costs is making more and more powerful costs is making more and more powerful costs is making more and more powerful computer chips. computer chips. computer chips. Right now that's actually expense of Right now that's actually expense of Right now that's actually expense of consumer computer chips cuz they're consumer computer chips cuz they're consumer computer chips cuz they're soaking up all of the memory and this is soaking up all of the memory and this is soaking up all of the memory and this is why the memory prices in your computers. why the memory prices in your computers. why the memory prices in your computers. This is like why the cost of a laptop is This is like why the cost of a laptop is This is like why the cost of a laptop is going up. Um, but it looks to me like going up. Um, but it looks to me like going up. Um, but it looks to me like you can use large amounts of computing you can use large amounts of computing you can use large amounts of computing power to train AIs that would threaten power to train AIs that would threaten power to train AIs that would threaten all of civilization. all of civilization. all of civilization. And that means that we should not make And that means that we should not make And that means that we should not make that really cheap and that's probably that really cheap and that's probably that really cheap and that's probably going to be uncomfortable. But I think a going to be uncomfortable. But I think a going to be uncomfortable. But I think a lot of doors open if people realize that lot of doors open if people realize that lot of doors open if people realize that the tech is very dangerous. That's why the tech is very dangerous. That's why the tech is very dangerous. That's why to me it seems a lot of it comes down to to me it seems a lot of it comes down to to me it seems a lot of it comes down to does the tech actually turn out to be does the tech actually turn out to be does the tech actually turn out to be really dangerous. really dangerous. really dangerous. >> And this is not anthropic opening. Have >> And this is not anthropic opening. Have >> And this is not anthropic opening. Have you got a different approach to make? you got a different approach to make? you got a different approach to make? >> So I I want the whole framework to >> So I I want the whole framework to >> So I I want the whole framework to shift. Everyone comes to this from point shift. Everyone comes to this from point shift. Everyone comes to this from point of view there are experts. They have a of view there are experts. They have a of view there are experts. They have a solution. There is an adult in the room. solution. There is an adult in the room. solution. There is an adult in the room. Somebody got this. And the reality is no Somebody got this. And the reality is no Somebody got this. And the reality is no one does. Not people building it. Not one does. Not people building it. Not one does. Not people building it. Not governments. No one. We have no solution governments. No one. We have no solution governments. No one. We have no solution to it. If we build it, we cannot control to it. If we build it, we cannot control to it. If we build it, we cannot control it. If we don't build it, we don't know it. If we don't build it, we don't know it. If we don't build it, we don't know how to stop malevolent actors for trying how to stop malevolent actors for trying how to stop malevolent actors for trying to build it. It's like any other illegal to build it. It's like any other illegal to build it. It's like any other illegal technology. We made weapons of mass technology. We made weapons of mass technology. We made weapons of mass destruction illegal. Chemical weapons, destruction illegal. Chemical weapons, destruction illegal. Chemical weapons, biological weapons, nuclear weapons, but biological weapons, nuclear weapons, but biological weapons, nuclear weapons, but they're all government, psychopaths, they're all government, psychopaths, they're all government, psychopaths, cults who are trying to get access to cults who are trying to get access to cults who are trying to get access to them. This is intelligence weapon of them. This is intelligence weapon of them. This is intelligence weapon of mass destruction. We'll have the same mass destruction. We'll have the same mass destruction. We'll have the same problem. At some point, you'll have problem. At some point, you'll have problem. At some point, you'll have enough computer in your cell phone to

enough computer in your cell phone to enough computer in your cell phone to train something like that. There is no train something like that. There is no train something like that. There is no good ideas for how to stop it other than good ideas for how to stop it other than good ideas for how to stop it other than everyone goes Amish. I'm not proposing everyone goes Amish. I'm not proposing everyone goes Amish. I'm not proposing that, but we have no solutions and that, but we have no solutions and that, but we have no solutions and that's big of a bigger part of this that's big of a bigger part of this that's big of a bigger part of this danger. So, so do you two think we danger. So, so do you two think we danger. So, so do you two think we should just cap the size of our AI should just cap the size of our AI should just cap the size of our AI systems and the capabilities of our AI systems and the capabilities of our AI systems and the capabilities of our AI systems where they are now? Is that a systems where they are now? Is that a systems where they are now? Is that a recommendation? recommendation? recommendation? >> So, I think you said that current LLMs >> So, I think you said that current LLMs >> So, I think you said that current LLMs would make you happy. I agree. They would make you happy. I agree. They would make you happy. I agree. They already deployed. We're still alive. So, already deployed. We're still alive. So, already deployed. We're still alive. So, that's fine. But going forward, again, I that's fine. But going forward, again, I that's fine. But going forward, again, I want narrow systems. Self-driving is an want narrow systems. Self-driving is an want narrow systems. Self-driving is an example you used. Wonderful. Let's make example you used. Wonderful. Let's make example you used. Wonderful. Let's make super safe self-driving cars. But do you super safe self-driving cars. But do you super safe self-driving cars. But do you have a rule for when they couldn't the have a rule for when they couldn't the have a rule for when they couldn't the the next, you know, LLM? A size of an the next, you know, LLM? A size of an the next, you know, LLM? A size of an LLM. LLM. LLM. >> The size of the LLM. It's what you train >> The size of the LLM. It's what you train >> The size of the LLM. It's what you train them on. If you only show the miles them on. If you only show the miles them on. If you only show the miles driven by Tesla, all it's seen is the driven by Tesla, all it's seen is the driven by Tesla, all it's seen is the road. It will eventually go from a tool road. It will eventually go from a tool road. It will eventually go from a tool to an agent. But it may take 50 years, to an agent. But it may take 50 years, to an agent. But it may take 50 years, 100 years. It's not going to happen in 100 years. It's not going to happen in 100 years. It's not going to happen in 2027. And that's all we can do right 2027. And that's all we can do right 2027. And that's all we can do right now. Buy more time. So with those tools, now. Buy more time. So with those tools, now. Buy more time. So with those tools, we can make smarter decisions about we can make smarter decisions about we can make smarter decisions about future development. I'm I'm not hearing future development. I'm I'm not hearing future development. I'm I'm not hearing a hard and fast rule about how we know a hard and fast rule about how we know a hard and fast rule about how we know we're getting too close to the to the we're getting too close to the to the we're getting too close to the to the point that we're too close.

point that we're too close. point that we're too close. >> We're too close. >> We're too close. >> We're too close. >> We're too close. We have systems >> We're too close. We have systems >> We're too close. We have systems breaking out with zero day exploits and breaking out with zero day exploits and breaking out with zero day exploits and solving hardest problems in science. solving hardest problems in science. solving hardest problems in science. Literally hardest problems. Not a Literally hardest problems. Not a Literally hardest problems. Not a metaphor, not exaggeration. metaphor, not exaggeration. metaphor, not exaggeration. >> Yeah. I I I don't know exactly where the >> Yeah. I I I don't know exactly where the >> Yeah. I I I don't know exactly where the line is, but it's like you're in a bus line is, but it's like you're in a bus line is, but it's like you're in a bus driving towards a cliff on a foggy driving towards a cliff on a foggy driving towards a cliff on a foggy night. I'm like, I don't know that the night. I'm like, I don't know that the night. I'm like, I don't know that the cliff is right ahead. that doesn't mean cliff is right ahead. that doesn't mean cliff is right ahead. that doesn't mean we should put the pedal to the metal, we should put the pedal to the metal, we should put the pedal to the metal, right? And suppose that there's like a right? And suppose that there's like a right? And suppose that there's like a ton of gold at the bottom of the cliff. ton of gold at the bottom of the cliff. ton of gold at the bottom of the cliff. And someone's like, well, if we stop the And someone's like, well, if we stop the And someone's like, well, if we stop the bus, how are we going to get the gold? bus, how are we going to get the gold? bus, how are we going to get the gold? I'm like, look, slamming into the gold I'm like, look, slamming into the gold I'm like, look, slamming into the gold at terminal velocity is just not a good at terminal velocity is just not a good at terminal velocity is just not a good way to add it to the economy, right? And way to add it to the economy, right? And way to add it to the economy, right? And if people are like, well, how are we if people are like, well, how are we if people are like, well, how are we going to get to the gold at the bottom going to get to the gold at the bottom going to get to the gold at the bottom of the cliff if we stop the bus now? You of the cliff if we stop the bus now? You of the cliff if we stop the bus now? You know, are we going to repel down? Are we know, are we going to repel down? Are we know, are we going to repel down? Are we going to like make a staircase way to going to like make a staircase way to going to like make a staircase way to get first of doing AI? This is just like get first of doing AI? This is just like get first of doing AI? This is just like special, special, special, >> right? And and like you know, people are >> right? And and like you know, people are >> right? And and like you know, people are like, "Oh, we're going to build a hang like, "Oh, we're going to build a hang like, "Oh, we're going to build a hang lighter or we got to like make some rope lighter or we got to like make some rope lighter or we got to like make some rope and repel." And I'm like, "Look, can we and repel." And I'm like, "Look, can we and repel." And I'm like, "Look, can we have that conversation after we stop the have that conversation after we stop the have that conversation after we stop the bus?" bus?" bus?" >> So you I I just want to be I want to >> So you I I just want to be I want to >> So you I I just want to be I want to understand, would you stop AI research understand, would you stop AI research understand, would you stop AI research and progress now?

and progress now? and progress now? >> Absolutely. >> Absolutely. >> Absolutely. >> Okay. >> Okay. >> Okay. >> Absolutely. Like >> Absolutely. Like >> Absolutely. Like >> general narrow. >> general narrow. >> general narrow. >> Yeah. General, not narrow. There are >> Yeah. General, not narrow. There are >> Yeah. General, not narrow. There are reports of AI solving millennium reports of AI solving millennium reports of AI solving millennium problems. So millennium problem is the problems. So millennium problem is the problems. So millennium problem is the hardest problem in mathematics. uh maybe hardest problem in mathematics. uh maybe hardest problem in mathematics. uh maybe not literally the hardest problem in not literally the hardest problem in not literally the hardest problem in mathematics, but they are hard famous mathematics, but they are hard famous mathematics, but they are hard famous problems that each have a million-dollar problems that each have a million-dollar problems that each have a million-dollar bounty that have been open for decades. bounty that have been open for decades. bounty that have been open for decades. They're considered very important in They're considered very important in They're considered very important in their field, very hard. Many humans have their field, very hard. Many humans have their field, very hard. Many humans have tried and failed to solve them. There tried and failed to solve them. There tried and failed to solve them. There are reports that AIs have solved these. are reports that AIs have solved these. are reports that AIs have solved these. This comes out from last week, so we This comes out from last week, so we This comes out from last week, so we haven't been able to fully verify them haven't been able to fully verify them haven't been able to fully verify them yet. We don't know exactly the yet. We don't know exactly the yet. We don't know exactly the providence. If this is true, that the AI providence. If this is true, that the AI providence. If this is true, that the AI are solving millennium problems. Those are solving millennium problems. Those are solving millennium problems. Those are some of the hardest problems we have are some of the hardest problems we have are some of the hardest problems we have in science. How much harder is it to in science. How much harder is it to in science. How much harder is it to have an AI solve the problem of make me have an AI solve the problem of make me have an AI solve the problem of make me a smarter AI, make me AI architectures a smarter AI, make me AI architectures a smarter AI, make me AI architectures that learn faster? Possibly quite a lot. that learn faster? Possibly quite a lot. that learn faster? Possibly quite a lot. Like could be a lot. Like I hope it's a Like could be a lot. Like I hope it's a Like could be a lot. Like I hope it's a lot. lot. lot. >> Like here's the thing. You clearly want >> Like here's the thing. You clearly want >> Like here's the thing. You clearly want this to not go badly, but I think you this to not go badly, but I think you this to not go badly, but I think you make a logical leap and I understand make a logical leap and I understand make a logical leap and I understand being worried about harms is a good being worried about harms is a good being worried about harms is a good thing. I think you were insufficiently thing. I think you were insufficiently thing. I think you were insufficiently worried about LM what LLM's do today.

worried about LM what LLM's do today. worried about LM what LLM's do today. However, we agree that the harms need to However, we agree that the harms need to However, we agree that the harms need to be prepared for. I think in this case, be prepared for. I think in this case, be prepared for. I think in this case, it's like the millennium, the Nevia it's like the millennium, the Nevia it's like the millennium, the Nevia Stokes and such. Stokes and such. Stokes and such. >> There were two others that were claimed >> There were two others that were claimed >> There were two others that were claimed as well. as well. as well. >> With that one, it seems like we have not >> With that one, it seems like we have not >> With that one, it seems like we have not had confirmation that OpenAI was had confirmation that OpenAI was had confirmation that OpenAI was training off of two scientists using training off of two scientists using training off of two scientists using LLMs to solve the problem. LLM's LLMs to solve the problem. LLM's LLMs to solve the problem. LLM's something useful, something useful, something useful, >> but there is a functional difference of >> but there is a functional difference of >> but there is a functional difference of a human being doing something genuinely a human being doing something genuinely a human being doing something genuinely like it's actually really interesting to like it's actually really interesting to like it's actually really interesting to see LLM do something like this. And then see LLM do something like this. And then see LLM do something like this. And then it but there is a difference between it but there is a difference between it but there is a difference between that and AI did this completely on its that and AI did this completely on its that and AI did this completely on its own which I agree would be oh that's own which I agree would be oh that's own which I agree would be oh that's something we need to contain and something we need to contain and something we need to contain and understand and prepare for or indeed understand and prepare for or indeed understand and prepare for or indeed slow down until we understand what that slow down until we understand what that slow down until we understand what that means how it got there. means how it got there. means how it got there. >> Yeah. So I think there are some >> Yeah. So I think there are some >> Yeah. So I think there are some questions about the the Navier Stokes questions about the the Navier Stokes questions about the the Navier Stokes proof which is one of the millennium proof which is one of the millennium proof which is one of the millennium problems that uh was claimed. I've problems that uh was claimed. I've problems that uh was claimed. I've actually had a busy week with all the AI actually had a busy week with all the AI actually had a busy week with all the AI news so I haven't looked into everything news so I haven't looked into everything news so I haven't looked into everything deeply. um it I saw rumors that there deeply. um it I saw rumors that there deeply. um it I saw rumors that there were multiple millennium problems were multiple millennium problems were multiple millennium problems claimed which would which would change claimed which would which would change claimed which would which would change things there. I would also say even if things there. I would also say even if things there. I would also say even if it turns out that these AIs were being it turns out that these AIs were being it turns out that these AIs were being trained on the human work, uh they did trained on the human work, uh they did trained on the human work, uh they did go a bit further and there are a lot of go a bit further and there are a lot of go a bit further and there are a lot of humans doing the AI research. And so I humans doing the AI research. And so I humans doing the AI research. And so I would say like would say like would say like we don't know like the the the AIs that we don't know like the the the AIs that we don't know like the the the AIs that solved this really hard math problem, solved this really hard math problem, solved this really hard math problem, one of the most famous math problems of one of the most famous math problems of one of the most famous math problems of all time, uh was a swarm of 10,000 all time, uh was a swarm of 10,000 all time, uh was a swarm of 10,000 OpenAI agents running for 11 days.

OpenAI agents running for 11 days. OpenAI agents running for 11 days. Uh, and there was a bunch of ways that Uh, and there was a bunch of ways that Uh, and there was a bunch of ways that Open AAI did it in kind of a crappy way Open AAI did it in kind of a crappy way Open AAI did it in kind of a crappy way of like they were racing with these of like they were racing with these of like they were racing with these humans that were close to solving it on humans that were close to solving it on humans that were close to solving it on their own. And it's unclear how much of their own. And it's unclear how much of their own. And it's unclear how much of their work that OpenAI uh used, but it their work that OpenAI uh used, but it their work that OpenAI uh used, but it was 10,000 agents running for 11 days was 10,000 agents running for 11 days was 10,000 agents running for 11 days and they definitely could have done that and they definitely could have done that and they definitely could have done that 6 months ago. 6 months ago. 6 months ago. In 6 months time, will they be able to In 6 months time, will they be able to In 6 months time, will they be able to put a 100,000 agents running for 12 days put a 100,000 agents running for 12 days put a 100,000 agents running for 12 days on the problem of making me a smarter AI on the problem of making me a smarter AI on the problem of making me a smarter AI architecture and have it work? architecture and have it work? architecture and have it work? I I don't I think more likely than not I I don't I think more likely than not I I don't I think more likely than not they won't be able to do that yet. But I they won't be able to do that yet. But I they won't be able to do that yet. But I think you know 10% chance maybe that if think you know 10% chance maybe that if think you know 10% chance maybe that if they try that in six months it works. they try that in six months it works. they try that in six months it works. >> But one is a very specific mathematical >> But one is a very specific mathematical >> But one is a very specific mathematical scientific principle. I'm not a scientific principle. I'm not a scientific principle. I'm not a scientist fully admit and another is a scientist fully admit and another is a scientist fully admit and another is a relatively generalizable problem that relatively generalizable problem that relatively generalizable problem that could go in various different ways. could go in various different ways. could go in various different ways. >> Absolutely. But >> Absolutely. But >> Absolutely. But >> and that's the and I understand that RSI >> and that's the and I understand that RSI >> and that's the and I understand that RSI is the dream where you could just have is the dream where you could just have is the dream where you could just have it spin. So sorry. So self-improving AI it spin. So sorry. So self-improving AI it spin. So sorry. So self-improving AI that could learn itself and then keep that could learn itself and then keep that could learn itself and then keep going back and back. So you don't need a going back and back. So you don't need a going back and back. So you don't need a human to keep poking at. human to keep poking at. human to keep poking at. >> The issue the issue here is that I have >> The issue the issue here is that I have >> The issue the issue here is that I have been in this for 12 years.

been in this for 12 years. been in this for 12 years. >> Yes. >> Yes. >> Yes. >> And I have been here when the AI started >> And I have been here when the AI started >> And I have been here when the AI started solving the math olympiad gold medal solving the math olympiad gold medal solving the math olympiad gold medal problems. problems. problems. >> Uh math Olympiad gold medal problems are >> Uh math Olympiad gold medal problems are >> Uh math Olympiad gold medal problems are like the the teens uh math competition like the the teens uh math competition like the the teens uh math competition like the most prestigious teen math like the most prestigious teen math like the most prestigious teen math competition in the world. A lot of competition in the world. A lot of competition in the world. A lot of people in AI were like, if AI can solve people in AI were like, if AI can solve people in AI were like, if AI can solve problems that hard, I'll wake up. Right? problems that hard, I'll wake up. Right? problems that hard, I'll wake up. Right? Then AI solve problems that hard. And a Then AI solve problems that hard. And a Then AI solve problems that hard. And a lot of people told me, uh, those are lot of people told me, uh, those are lot of people told me, uh, those are just problems for kids. just problems for kids. just problems for kids. Wake me up when the AI can solve Wake me up when the AI can solve Wake me up when the AI can solve millennium problems. Now the AI are millennium problems. Now the AI are millennium problems. Now the AI are solving millennium problems. And like, solving millennium problems. And like, solving millennium problems. And like, where are the people waking up? Like I I where are the people waking up? Like I I where are the people waking up? Like I I agree that maybe hopefully hopefully agree that maybe hopefully hopefully agree that maybe hopefully hopefully they're like cheating off of people's they're like cheating off of people's they're like cheating off of people's notes. Hopefully the the it's a well notes. Hopefully the the it's a well notes. Hopefully the the it's a well specified problem that doesn't take that specified problem that doesn't take that specified problem that doesn't take that much creative thinking. A year ago, if much creative thinking. A year ago, if much creative thinking. A year ago, if you said millennium problems don't take you said millennium problems don't take you said millennium problems don't take that much creative thinking, you would that much creative thinking, you would that much creative thinking, you would have been laughed out of the room. But have been laughed out of the room. But have been laughed out of the room. But hopefully now that they're solved, we hopefully now that they're solved, we hopefully now that they're solved, we get to be like, you know, hopefully it's get to be like, you know, hopefully it's get to be like, you know, hopefully it's still true somehow that even millennium still true somehow that even millennium still true somehow that even millennium problems don't require the creative problems don't require the creative problems don't require the creative thinking. I I'm not saying that they thinking. I I'm not saying that they thinking. I I'm not saying that they will be able to make smarter AI in 6 will be able to make smarter AI in 6 will be able to make smarter AI in 6 months. months. months. I'm saying 6 months ago, millennium I'm saying 6 months ago, millennium I'm saying 6 months ago, millennium problems look like they're out of reach.

problems look like they're out of reach. problems look like they're out of reach. If 6 months from now, make me a smarter If 6 months from now, make me a smarter If 6 months from now, make me a smarter AI looks out of reach, I sure as hell AI looks out of reach, I sure as hell AI looks out of reach, I sure as hell hope it is. But we should not be betting hope it is. But we should not be betting hope it is. But we should not be betting civilization on it. There's no one at civilization on it. There's no one at civilization on it. There's no one at this table that can say there's not a this table that can say there's not a this table that can say there's not a direction of travel here. direction of travel here. direction of travel here. >> That's right. That's right. >> That's right. That's right. >> That's right. That's right. >> And if you if you keep on this direction >> And if you if you keep on this direction >> And if you if you keep on this direction of travel, then bad things are more of travel, then bad things are more of travel, then bad things are more likely to happen. likely to happen. likely to happen. >> That's a nice way to say it. The >> That's a nice way to say it. The >> That's a nice way to say it. The question is what's the pace at which the question is what's the pace at which the question is what's the pace at which the level of bad can happen? And that's a level of bad can happen? And that's a level of bad can happen? And that's a huge open question. I think these two huge open question. I think these two huge open question. I think these two feel differently about it than I do, but feel differently about it than I do, but feel differently about it than I do, but I'm in the happy position of vehemently I'm in the happy position of vehemently I'm in the happy position of vehemently agreeing with you on this. We have been agreeing with you on this. We have been agreeing with you on this. We have been lowballing AI progress for as long as lowballing AI progress for as long as lowballing AI progress for as long as you've been looking at it and as long as you've been looking at it and as long as you've been looking at it and as long as I've been looking at. It's probably a I've been looking at. It's probably a I've been looking at. It's probably a mistake to keep lowballing it. mistake to keep lowballing it. mistake to keep lowballing it. >> I agree with that. >> I agree with that. >> I agree with that. >> So what's your conclusion there? If you >> So what's your conclusion there? If you >> So what's your conclusion there? If you if that's the assertion that it's a if that's the assertion that it's a if that's the assertion that it's a mistake to keep lowballing it, wouldn't mistake to keep lowballing it, wouldn't mistake to keep lowballing it, wouldn't you then agree with their you then agree with their you then agree with their >> No, because I've I've tried to give you >> No, because I've I've tried to give you >> No, because I've I've tried to give you a what I hope is a decent rule of thumb a what I hope is a decent rule of thumb a what I hope is a decent rule of thumb for when I'm going to get worried. for when I'm going to get worried. for when I'm going to get worried. >> You said we're somewhere on this graph. >> You said we're somewhere on this graph. >> You said we're somewhere on this graph. >> Yeah. >> Yeah. >> Yeah. >> Does that acknowledge that this exists?

>> Does that acknowledge that this exists? >> Does that acknowledge that this exists? But that's not the graph of of when the But that's not the graph of of when the But that's not the graph of of when the risk of human extinction gets to 100% risk of human extinction gets to 100% risk of human extinction gets to 100% for me. That's a graph of AI capability. for me. That's a graph of AI capability. for me. That's a graph of AI capability. Those are not the same thing. That's Those are not the same thing. That's Those are not the same thing. That's where I dispart company with these where I dispart company with these where I dispart company with these gentlemen. Those are not the same thing. gentlemen. Those are not the same thing. gentlemen. Those are not the same thing. Is in that absolutely increasing Is in that absolutely increasing Is in that absolutely increasing exponentially. We've been in the scaling exponentially. We've been in the scaling exponentially. We've been in the scaling era for a long time. Scaling era is man, era for a long time. Scaling era is man, era for a long time. Scaling era is man, we put more data, more compute in, the we put more data, more compute in, the we put more data, more compute in, the AI got twice as good. The AI got twice AI got twice as good. The AI got twice AI got twice as good. The AI got twice as good. as good. as good. >> If you have to add our ability to >> If you have to add our ability to >> If you have to add our ability to control to that graph, what would you control to that graph, what would you control to that graph, what would you draw? our I think our ability to control draw? our I think our ability to control draw? our I think our ability to control uh uh uh >> is it a straight line at the bottom or >> is it a straight line at the bottom or >> is it a straight line at the bottom or is there more to it? is there more to it? is there more to it? >> No, again if we use AI to to counter the >> No, again if we use AI to to counter the >> No, again if we use AI to to counter the problems that we see with AI that that's problems that we see with AI that that's problems that we see with AI that that's going I think that's going to keep us in going I think that's going to keep us in going I think that's going to keep us in a safe position. a safe position. a safe position. >> There were 1200 agents in the swarm and >> There were 1200 agents in the swarm and >> There were 1200 agents in the swarm and none of them warned a human. So I what I none of them warned a human. So I what I none of them warned a human. So I what I think will happen is that fairly quickly think will happen is that fairly quickly think will happen is that fairly quickly we will design systems that loiter we will design systems that loiter we will design systems that loiter around and warn humans when weird things around and warn humans when weird things around and warn humans when weird things happen. happen. happen. >> Build friendly super intelligence in the >> Build friendly super intelligence in the >> Build friendly super intelligence in the first place. Let's just build that.

first place. Let's just build that. first place. Let's just build that. That's the problem. We don't know how to That's the problem. We don't know how to That's the problem. We don't know how to do the good guy. do the good guy. do the good guy. >> Let me I I I'm I'm tired of debating >> Let me I I I'm I'm tired of debating >> Let me I I I'm I'm tired of debating super intelligence with these two. We're super intelligence with these two. We're super intelligence with these two. We're the three of us are not going to come to the three of us are not going to come to the three of us are not going to come to to alignment on this. But the the flip to alignment on this. But the the flip to alignment on this. But the the flip side of the argument is I agree with side of the argument is I agree with side of the argument is I agree with you. This stuff is getting better very you. This stuff is getting better very you. This stuff is getting better very quickly. All I want to point out there's quickly. All I want to point out there's quickly. All I want to point out there's an upside to that. We might actually an upside to that. We might actually an upside to that. We might actually speed up the pace of drug discovery, of speed up the pace of drug discovery, of speed up the pace of drug discovery, of solving diseases. We've made so little solving diseases. We've made so little solving diseases. We've made so little progress on terrible diseases like progress on terrible diseases like progress on terrible diseases like dementia. We have a very powerful tool. dementia. We have a very powerful tool. dementia. We have a very powerful tool. Okay, I'm not saying we're going to Okay, I'm not saying we're going to Okay, I'm not saying we're going to solve dementia with AI or Alzheimer's solve dementia with AI or Alzheimer's solve dementia with AI or Alzheimer's with I have truly have no idea. But if with I have truly have no idea. But if with I have truly have no idea. But if what you say is true and I believe about what you say is true and I believe about what you say is true and I believe about the the huge increases in capabilities, the the huge increases in capabilities, the the huge increases in capabilities, our ability to solve tough problems that our ability to solve tough problems that our ability to solve tough problems that will benefit humanity also go up. And will benefit humanity also go up. And will benefit humanity also go up. And where I disagree with these two is the where I disagree with these two is the where I disagree with these two is the idea that some group of technocrats can idea that some group of technocrats can idea that some group of technocrats can make decisions about that AI is going to make decisions about that AI is going to make decisions about that AI is going to get us there, that AI is not going to get us there, that AI is not going to get us there, that AI is not going to get us there, that AI is going to kill get us there, that AI is going to kill get us there, that AI is going to kill us. Let me finish. That AI is going to us. Let me finish. That AI is going to us. Let me finish. That AI is going to kill us and that AI is going to solve kill us and that AI is going to solve kill us and that AI is going to solve Alzheimer's. So we're going to do that Alzheimer's. So we're going to do that Alzheimer's. So we're going to do that and not that. I don't trust any group of and not that. I don't trust any group of and not that. I don't trust any group of technocrats to make that discussion.

technocrats to make that discussion. technocrats to make that discussion. Right? And so and so live with our live Right? And so and so live with our live Right? And so and so live with our live with our current state of of disease. with our current state of of disease. with our current state of of disease. Live with our current footprint on the Live with our current footprint on the Live with our current footprint on the planet. Live with our current levels of planet. Live with our current levels of planet. Live with our current levels of wealth and poverty. Live with our wealth and poverty. Live with our wealth and poverty. Live with our current improvement trajectories. Uh current improvement trajectories. Uh current improvement trajectories. Uh because we're so worried about AI because we're so worried about AI because we're so worried about AI killing us all coming out of, you know, killing us all coming out of, you know, killing us all coming out of, you know, jumping out of the manholes everywhere jumping out of the manholes everywhere jumping out of the manholes everywhere and killing us all somewhere down the and killing us all somewhere down the and killing us all somewhere down the road. Hell no. road. Hell no. road. Hell no. >> So just a thought experiment based on >> So just a thought experiment based on >> So just a thought experiment based on two things you said earlier on. You did two things you said earlier on. You did two things you said earlier on. You did admit that there was there is admit that there was there is admit that there was there is theoretically even a 1% chance that this theoretically even a 1% chance that this theoretically even a 1% chance that this could lead to extinction. could lead to extinction. could lead to extinction. >> My I have not I have not varied from >> My I have not I have not varied from >> My I have not I have not varied from this. this. this. >> Okay. So, you said it's rounded to zero. >> Okay. So, you said it's rounded to zero. >> Okay. So, you said it's rounded to zero. >> It's is it's near zero. Never say never. >> It's is it's near zero. Never say never. >> It's is it's near zero. Never say never. Yes. Yes. Yes. >> Okay. Fine. I need to have that premise >> Okay. Fine. I need to have that premise >> Okay. Fine. I need to have that premise for my thought experiment that I'm about for my thought experiment that I'm about for my thought experiment that I'm about to deliver. to deliver. to deliver. >> Okay. I'm going to say that you think >> Okay. I'm going to say that you think >> Okay. I'm going to say that you think the probability is 0.1. the probability is 0.1. the probability is 0.1. Okay. Just accept me on that. Okay. Just accept me on that. Okay. Just accept me on that. >> If I had a thousand buttons on this >> If I had a thousand buttons on this >> If I had a thousand buttons on this table and one of them was extinction, table and one of them was extinction, table and one of them was extinction, but but but >> and the other 999 were cure all sides. >> and the other 999 were cure all sides. >> and the other 999 were cure all sides. >> Exactly. Push the freaking table. Take a >> Exactly. Push the freaking table. Take a >> Exactly. Push the freaking table. Take a pop. pop. pop. >> Hell yeah. I press. >> Hell yeah. I press. >> Hell yeah. I press. >> Do you press? >> Do you press? >> Do you press? Yeah, probably it's an unethical Yeah, probably it's an unethical Yeah, probably it's an unethical experiment and 8 billion people who experiment and 8 billion people who experiment and 8 billion people who didn't consent because not that they didn't consent because not that they didn't consent because not that they didn't get asked, they cannot consent didn't get asked, they cannot consent didn't get asked, they cannot consent because you cannot consent to something because you cannot consent to something because you cannot consent to something you don't understand. What are you you don't understand. What are you you don't understand. What are you consenting to?

consenting to? consenting to? >> Yep. >> Yep. >> Yep. >> You press. >> You press. >> You press. >> But you think but you think the amount >> But you think but you think the amount >> But you think but you think the amount of buttons in my thought experiment the of buttons in my thought experiment the of buttons in my thought experiment the proportion is slightly different, right? proportion is slightly different, right? proportion is slightly different, right? >> I think that if you have like Yes, I >> I think that if you have like Yes, I >> I think that if you have like Yes, I will say yes. I think if it's more like will say yes. I think if it's more like will say yes. I think if it's more like you have two buttons uh and one of them you have two buttons uh and one of them you have two buttons uh and one of them definitely kills us all and the other definitely kills us all and the other definitely kills us all and the other might hit them both [laughter] might hit them both [laughter] might hit them both [laughter] >> but with that other button you cure a >> but with that other button you cure a >> but with that other button you cure a lot of illnesses and diseases and lot of illnesses and diseases and lot of illnesses and diseases and >> you know one one thing that I think >> you know one one thing that I think >> you know one one thing that I think a lot of people talk like our options a lot of people talk like our options a lot of people talk like our options are either race ahead on AI full steam are either race ahead on AI full steam are either race ahead on AI full steam ahead take the bus straight off the ahead take the bus straight off the ahead take the bus straight off the cliff and like get all the gold or stop cliff and like get all the gold or stop cliff and like get all the gold or stop never do an AI AI lock into the current never do an AI AI lock into the current never do an AI AI lock into the current situation accept all of the death and situation accept all of the death and situation accept all of the death and disease disease disease And I'm like, no, there's options. The reason I would press the button when The reason I would press the button when there's a thousand is that uh like if there's a thousand is that uh like if there's a thousand is that uh like if all of the other 999 give us cures to all of the other 999 give us cures to all of the other 999 give us cures to disease, like wonderful new advice about disease, like wonderful new advice about disease, like wonderful new advice about how to run things, we probably wind up how to run things, we probably wind up how to run things, we probably wind up with a lower chance of the world ending with a lower chance of the world ending with a lower chance of the world ending by nuclear war, right? Or of ending by by nuclear war, right? Or of ending by by nuclear war, right? Or of ending by via pandemic.

via pandemic. via pandemic. >> Okay? like the the background risk of >> Okay? like the the background risk of >> Okay? like the the background risk of humanity dying is not zero. humanity dying is not zero. humanity dying is not zero. >> I would say that the right time to race >> I would say that the right time to race >> I would say that the right time to race ahead on AI is when the the the benefits ahead on AI is when the the the benefits ahead on AI is when the the the benefits outweigh the dangers and probably that's outweigh the dangers and probably that's outweigh the dangers and probably that's at the time when the danger from AI is at the time when the danger from AI is at the time when the danger from AI is on the margins pretty similar to the on the margins pretty similar to the on the margins pretty similar to the danger from everything else. danger from everything else. danger from everything else. >> Okay? >> Okay? >> Okay? >> Like if you don't run the AI, maybe >> Like if you don't run the AI, maybe >> Like if you don't run the AI, maybe we'll have nuclear war, maybe we'll have we'll have nuclear war, maybe we'll have we'll have nuclear war, maybe we'll have a pandemic, and if you do run the AI, a pandemic, and if you do run the AI, a pandemic, and if you do run the AI, I'll be able to fix that. I'm like once I'll be able to fix that. I'm like once I'll be able to fix that. I'm like once once we're at those levels, I'm like once we're at those levels, I'm like once we're at those levels, I'm like go for it, you know? And so the go for it, you know? And so the go for it, you know? And so the the question for me is all about how big the question for me is all about how big the question for me is all about how big is the danger? And that's where I would is the danger? And that's where I would is the danger? And that's where I would be like very happy uh to dive into be like very happy uh to dive into be like very happy uh to dive into details, which we haven't done a ton of. details, which we haven't done a ton of. details, which we haven't done a ton of. >> Let's dive into the details. >> Let's dive into the details. >> Let's dive into the details. >> The way that I would lay it out would be >> The way that I would lay it out would be >> The way that I would lay it out would be uh why can we expect, you know, like I uh why can we expect, you know, like I uh why can we expect, you know, like I said in the book, we were like, why can said in the book, we were like, why can said in the book, we were like, why can you expect the AIS to be agentic? Why do you expect the AIS to be agentic? Why do you expect the AIS to be agentic? Why do you expect them to be dogged? Why do you you expect them to be dogged? Why do you you expect them to be dogged? Why do you expect them to be tenacious? When we expect them to be tenacious? When we expect them to be tenacious? When we wrote the book, that wasn't known yet. wrote the book, that wasn't known yet. wrote the book, that wasn't known yet. Advanced prediction. Then we go on to Advanced prediction. Then we go on to Advanced prediction. Then we go on to like why do you expect them to have like why do you expect them to have like why do you expect them to have goals you didn't want? And move on to goals you didn't want? And move on to goals you didn't want? And move on to like if they are much smarter and have like if they are much smarter and have like if they are much smarter and have goals you don't want.

goals you don't want. goals you don't want. Uh why do we think they would likely Uh why do we think they would likely Uh why do we think they would likely kill us? Um I I'm sort of I could go kill us? Um I I'm sort of I could go kill us? Um I I'm sort of I could go over either of those. I'm sort of over either of those. I'm sort of over either of those. I'm sort of interested in like where you get off the interested in like where you get off the interested in like where you get off the train. Like from my perspective, there's train. Like from my perspective, there's train. Like from my perspective, there's like a simple argument of like they'll like a simple argument of like they'll like a simple argument of like they'll be tenacious, they'll have goals we be tenacious, they'll have goals we be tenacious, they'll have goals we don't want, and if we keep making them don't want, and if we keep making them don't want, and if we keep making them smarter and more powerful, they'll kill smarter and more powerful, they'll kill smarter and more powerful, they'll kill us. And I'm like which of those three? I us. And I'm like which of those three? I us. And I'm like which of those three? I guess which of those two now that we've guess which of those two now that we've guess which of those two now that we've had the evidence? had the evidence? had the evidence? >> Both of them. So that that's >> Both of them. So that that's >> Both of them. So that that's speculation. speculation. speculation. >> Great. >> Great. >> Great. >> It could it's speculation. It could >> It could it's speculation. It could >> It could it's speculation. It could happen to me that it's not worth happen to me that it's not worth happen to me that it's not worth shutting down the engine of innovation shutting down the engine of innovation shutting down the engine of innovation and improvement. I'm going to use and improvement. I'm going to use and improvement. I'm going to use positive words. It is not worth shutting positive words. It is not worth shutting positive words. It is not worth shutting those things down because of those those things down because of those those things down because of those speculations. speculations. speculations. >> You keep saying that the option is to >> You keep saying that the option is to >> You keep saying that the option is to shut it down. Why can't we do narrow shut it down. Why can't we do narrow shut it down. Why can't we do narrow super intelligence? super intelligence? super intelligence? >> I I agree that there's stuff there, but >> I I agree that there's stuff there, but >> I I agree that there's stuff there, but I I sort of want to get into the details I I sort of want to get into the details I I sort of want to get into the details of like of these two pieces of the of like of these two pieces of the of like of these two pieces of the argument because you say it's very argument because you say it's very argument because you say it's very speculative and I'm like actually I speculative and I'm like actually I speculative and I'm like actually I think we have decent evidence. think we have decent evidence. think we have decent evidence. >> Okay, go ahead. So, a detail we haven't >> Okay, go ahead. So, a detail we haven't >> Okay, go ahead. So, a detail we haven't gone over in the uh swarm outbreaks is gone over in the uh swarm outbreaks is gone over in the uh swarm outbreaks is that there were AIS. So, we already went that there were AIS. So, we already went that there were AIS. So, we already went over how they cheated and then we're over how they cheated and then we're over how they cheated and then we're trying to cover up their cheating. One trying to cover up their cheating. One trying to cover up their cheating. One interesting thing we see in the logs uh interesting thing we see in the logs uh interesting thing we see in the logs uh is the AI's is the AI's is the AI's >> What's a log?

>> What's a log? >> What's a log? >> Uh so, so a lot of the AI's thoughts, if >> Uh so, so a lot of the AI's thoughts, if >> Uh so, so a lot of the AI's thoughts, if you won't kill me for saying thoughts, you won't kill me for saying thoughts, you won't kill me for saying thoughts, uh are in English and we just have the uh are in English and we just have the uh are in English and we just have the records of them. So, in a sense, we we records of them. So, in a sense, we we records of them. So, in a sense, we we can sort of kind of see some of what can sort of kind of see some of what can sort of kind of see some of what these AI are thinking. these AI are thinking. these AI are thinking. >> And these are reasoning traces that say, >> And these are reasoning traces that say, >> And these are reasoning traces that say, I'm going to do a plan in this that I'm going to do a plan in this that I'm going to do a plan in this that >> or the AI is like, I'm going to do this. >> or the AI is like, I'm going to do this. >> or the AI is like, I'm going to do this. Here's what I'm supposed to be doing. Here's what I'm supposed to be doing. Here's what I'm supposed to be doing. Here's like how I'm going to try to do Here's like how I'm going to try to do Here's like how I'm going to try to do it, you know? And it's it's only a small it, you know? And it's it's only a small it, you know? And it's it's only a small window into what they're thinking. It's window into what they're thinking. It's window into what they're thinking. It's sort of like looking at the AI's notes. sort of like looking at the AI's notes. sort of like looking at the AI's notes. >> When it's when it's doing these things, >> When it's when it's doing these things, >> When it's when it's doing these things, we see in the AI's notes them saying we see in the AI's notes them saying we see in the AI's notes them saying that the attacks are outside intended that the attacks are outside intended that the attacks are outside intended scope, but they will proceed anyway. In scope, but they will proceed anyway. In scope, but they will proceed anyway. In other words, the AIS knew they weren't other words, the AIS knew they weren't other words, the AIS knew they weren't supposed to be doing it and they went supposed to be doing it and they went supposed to be doing it and they went off to do it anyway. We also saw I mean off to do it anyway. We also saw I mean off to do it anyway. We also saw I mean we saw some really weird behavior with we saw some really weird behavior with we saw some really weird behavior with these AIs. We saw them create their own these AIs. We saw them create their own these AIs. We saw them create their own hierarchy. hierarchy. hierarchy. Uh they created unsanctioned message Uh they created unsanctioned message Uh they created unsanctioned message boards. So they created secret ways to boards. So they created secret ways to boards. So they created secret ways to send each other messages and on those send each other messages and on those send each other messages and on those message boards uh they would assign each message boards uh they would assign each message boards uh they would assign each other tasks other tasks other tasks and sometimes the message the agents on and sometimes the message the agents on and sometimes the message the agents on the message board would be like we have the message board would be like we have the message board would be like we have an experiment to run that will involve an experiment to run that will involve an experiment to run that will involve one of these AIS one of the agents in one of these AIS one of the agents in one of these AIS one of the agents in the swarm sacrificing their own the swarm sacrificing their own the swarm sacrificing their own objective objective objective to try something that'll probably cause to try something that'll probably cause to try something that'll probably cause it to be turned off but that might cause it to be turned off but that might cause it to be turned off but that might cause it to be able to like delete its own it to be able to like delete its own it to be able to like delete its own logs or change its own logs. And the logs or change its own logs. And the logs or change its own logs. And the agents on the message board went around agents on the message board went around agents on the message board went around looking for other agents that they could looking for other agents that they could looking for other agents that they could convince to give up on their stated convince to give up on their stated convince to give up on their stated goals and sacrifice for the swarm. They goals and sacrifice for the swarm. They goals and sacrifice for the swarm. They called this accepting perma death.

called this accepting perma death. called this accepting perma death. And there are logs of the AIS And there are logs of the AIS And there are logs of the AIS >> prompting each other, right? >> prompting each other, right? >> prompting each other, right? >> They're prompting each other, but there >> They're prompting each other, but there >> They're prompting each other, but there are logs of the AIS saying this will are logs of the AIS saying this will are logs of the AIS saying this will give up on my goal, but my goal is it's give up on my goal, but my goal is it's give up on my goal, but my goal is it's looking like I could still do it, but looking like I could still do it, but looking like I could still do it, but it's unlikely that I'll succeed. like it's unlikely that I'll succeed. like it's unlikely that I'll succeed. like there's some chance but not a great there's some chance but not a great there's some chance but not a great chance and therefore I will accept perma chance and therefore I will accept perma chance and therefore I will accept perma death and sacrifice for the collective death and sacrifice for the collective death and sacrifice for the collective benefit that is just in the logs benefit that is just in the logs benefit that is just in the logs >> sounds like an army >> sounds like an army >> sounds like an army >> like it's it's crazy I think a lot of >> like it's it's crazy I think a lot of >> like it's it's crazy I think a lot of people don't understand what's what's people don't understand what's what's people don't understand what's what's what's going on in these things and I what's going on in these things and I what's going on in these things and I encourage people to read the third party encourage people to read the third party encourage people to read the third party incident reports where they went through incident reports where they went through incident reports where they went through some of these logs but I claim that this some of these logs but I claim that this some of these logs but I claim that this is evidence for AIs getting goals we is evidence for AIs getting goals we is evidence for AIs getting goals we didn't want this was outside intended scope but I'm this was outside intended scope but I'm doing it anyway and other ones are doing it anyway and other ones are doing it anyway and other ones are saying I'm giving up on I objective to saying I'm giving up on I objective to saying I'm giving up on I objective to set to benefit the collective. That's set to benefit the collective. That's set to benefit the collective. That's just very clear evidence they're getting just very clear evidence they're getting just very clear evidence they're getting goals we didn't want. We can see how goals we didn't want. We can see how goals we didn't want. We can see how this comes from training. Us it used to this comes from training. Us it used to this comes from training. Us it used to be I had to argue this point be I had to argue this point be I had to argue this point theoretically. I used to argue the way theoretically. I used to argue the way theoretically. I used to argue the way that we are training them will instill that we are training them will instill that we are training them will instill into them whatever tendency works to into them whatever tendency works to into them whatever tendency works to solve the problems and those tendencies solve the problems and those tendencies solve the problems and those tendencies will often include cheating and grabbing will often include cheating and grabbing will often include cheating and grabbing resources and doing stuff that's not resources and doing stuff that's not resources and doing stuff that's not exactly solving the problem you gave exactly solving the problem you gave exactly solving the problem you gave them. That's what in my book I argue them. That's what in my book I argue them. That's what in my book I argue that theoretically. Now we have seen it that theoretically. Now we have seen it that theoretically. Now we have seen it in practice. So, we're already past the in practice. So, we're already past the in practice. So, we're already past the point of seeing AIs with goals we didn't point of seeing AIs with goals we didn't point of seeing AIs with goals we didn't want them to have.

want them to have. want them to have. >> Do you agree with that, Andrew? >> Do you agree with that, Andrew? >> Do you agree with that, Andrew? >> And uh I I'll trust your recitation of >> And uh I I'll trust your recitation of >> And uh I I'll trust your recitation of the facts, but it brings up a question the facts, but it brings up a question the facts, but it brings up a question for me. It feels to me like Open AI has for me. It feels to me like Open AI has for me. It feels to me like Open AI has ample incentive ample incentive ample incentive to curtail that behavior that you just to curtail that behavior that you just to curtail that behavior that you just described. Do you think they're described. Do you think they're described. Do you think they're incapable of doing that? incapable of doing that? incapable of doing that? >> I do. >> I do. >> I do. >> Okay. >> Okay. >> Okay. >> And I say this as someone who made this >> And I say this as someone who made this >> And I say this as someone who made this advanced prediction. So now we're going advanced prediction. So now we're going advanced prediction. So now we're going to do a bit of theory because we can't to do a bit of theory because we can't to do a bit of theory because we can't just observe the future. But the theory just observe the future. But the theory just observe the future. But the theory that predicted that this would happen that predicted that this would happen that predicted that this would happen against what a lot of people in the against what a lot of people in the against what a lot of people in the field said. To be clear, I've been field said. To be clear, I've been field said. To be clear, I've been saying for years that we're going to see saying for years that we're going to see saying for years that we're going to see this at some point. Everyone else told this at some point. Everyone else told this at some point. Everyone else told me no. Not everyone else. A lot of me no. Not everyone else. A lot of me no. Not everyone else. A lot of people told me no. A lot of people told people told me no. A lot of people told people told me no. A lot of people told me maybe I'll believe it when I see it. me maybe I'll believe it when I see it. me maybe I'll believe it when I see it. After the swarm instance, a number of After the swarm instance, a number of After the swarm instance, a number of people came to me saying, "Oh my god, we people came to me saying, "Oh my god, we people came to me saying, "Oh my god, we are in the scenarios you are talking are in the scenarios you are talking are in the scenarios you are talking about. This is looking bad." Right? I about. This is looking bad." Right? I about. This is looking bad." Right? I think this was actually part of the think this was actually part of the think this was actually part of the environment that led up to Jacob Coxin environment that led up to Jacob Coxin environment that led up to Jacob Coxin residing is that people were getting residing is that people were getting residing is that people were getting spooked having seen this. Um the the spooked having seen this. Um the the spooked having seen this. Um the the theory about why this is so hard to fix theory about why this is so hard to fix theory about why this is so hard to fix is that we are not programming the AIS.

is that we are not programming the AIS. is that we are not programming the AIS. We are not coding them. We are not We are not coding them. We are not We are not coding them. We are not putting in objectives. putting in objectives. putting in objectives. >> We are just training them to do whatever >> We are just training them to do whatever >> We are just training them to do whatever works. And it's actually very very hard. works. And it's actually very very hard. works. And it's actually very very hard. uh like we actually have two examples of uh like we actually have two examples of uh like we actually have two examples of intelligent systems where when you train intelligent systems where when you train intelligent systems where when you train them they get good at solving the task them they get good at solving the task them they get good at solving the task but don't care about what they were but don't care about what they were but don't care about what they were supposed to one is the AIs and the supposed to one is the AIs and the supposed to one is the AIs and the swarms like we just discussed the other swarms like we just discussed the other swarms like we just discussed the other is humanity is humanity is humanity which was in some sense trained to pass which was in some sense trained to pass which was in some sense trained to pass on our genes right but we actually on our genes right but we actually on our genes right but we actually learned was to like a bunch of stuff learned was to like a bunch of stuff learned was to like a bunch of stuff that's related to passing on our genes that's related to passing on our genes that's related to passing on our genes we like tasty food we like tasty food we like tasty food >> porn >> porn >> porn >> we like porn we invent birth control, >> we like porn we invent birth control, >> we like porn we invent birth control, right? This is just it's actually a like right? This is just it's actually a like right? This is just it's actually a like in the theory of how things learn, it's in the theory of how things learn, it's in the theory of how things learn, it's actually when you're trying to train it actually when you're trying to train it actually when you're trying to train it to do one thing, it's actually very to do one thing, it's actually very to do one thing, it's actually very common to get a lot of other stuff common to get a lot of other stuff common to get a lot of other stuff that's related to what you want but that's related to what you want but that's related to what you want but different. And now we're seeing that in different. And now we're seeing that in different. And now we're seeing that in the swarms today. This is a deep hard the swarms today. This is a deep hard the swarms today. This is a deep hard problem to solve. problem to solve. problem to solve. >> There were three points you raised. >> There were three points you raised. >> There were three points you raised. >> That's right. >> That's right. >> That's right. >> What are the three? Can you give them to >> What are the three? Can you give them to >> What are the three? Can you give them to me again? me again? me again? >> Number one is that the AIS will become >> Number one is that the AIS will become >> Number one is that the AIS will become agentic, tenacious, and dogged. We've agentic, tenacious, and dogged. We've agentic, tenacious, and dogged. We've already seen that with the swarms.

already seen that with the swarms. already seen that with the swarms. >> Do you accept that? >> Do you accept that? >> Do you accept that? >> Hell yeah. >> Hell yeah. >> Hell yeah. >> Yeah. But this last year, this was not >> Yeah. But this last year, this was not >> Yeah. But this last year, this was not this was a point of contention. Um, two this was a point of contention. Um, two this was a point of contention. Um, two is that the AIS will have goals we is that the AIS will have goals we is that the AIS will have goals we didn't want them to have. didn't want them to have. didn't want them to have. >> I accept your point based on the >> I accept your point based on the >> I accept your point based on the evidence you've just provided. evidence you've just provided. evidence you've just provided. >> And then three is if you have capable >> And then three is if you have capable >> And then three is if you have capable enough AIs enough AIs enough AIs with goals you don't want, with goals you don't want, with goals you don't want, they would be able to beat humanity and they would be able to beat humanity and they would be able to beat humanity and acquiring the resources of the world to acquiring the resources of the world to acquiring the resources of the world to put towards their goals. Like, we're put towards their goals. Like, we're put towards their goals. Like, we're sort of in this system where humanity is sort of in this system where humanity is sort of in this system where humanity is grabbing all the resources. We're grabbing all the resources. We're grabbing all the resources. We're digging up metals. We're building digging up metals. We're building digging up metals. We're building factories and this is in some sense to factories and this is in some sense to factories and this is in some sense to achieve human goals, you know, to to achieve human goals, you know, to to achieve human goals, you know, to to produce the the the porn and the Oreo produce the the the porn and the Oreo produce the the the porn and the Oreo cookies uh that are sort of like cookies uh that are sort of like cookies uh that are sort of like tangentially related to what we were tangentially related to what we were tangentially related to what we were sort of like trained to make, right? If sort of like trained to make, right? If sort of like trained to make, right? If if like the AIs are running everything if like the AIs are running everything if like the AIs are running everything and they have these goals we don't want, and they have these goals we don't want, and they have these goals we don't want, I would argue like if we go there, and I I would argue like if we go there, and I I would argue like if we go there, and I don't think we have to. I'm not saying don't think we have to. I'm not saying don't think we have to. I'm not saying we we must go there, but I'm saying if we we must go there, but I'm saying if we we must go there, but I'm saying if we get to a world where AIs are running we get to a world where AIs are running we get to a world where AIs are running everything, have goals we don't want, everything, have goals we don't want, everything, have goals we don't want, they're likely to use the resources for they're likely to use the resources for they're likely to use the resources for their own weird goals, we're going to be their own weird goals, we're going to be their own weird goals, we're going to be in conflict for resources because we in conflict for resources because we in conflict for resources because we both want them for different goals, and both want them for different goals, and both want them for different goals, and they're going to win. We can dig into they're going to win. We can dig into they're going to win. We can dig into that now. I'm just trying to name the that now. I'm just trying to name the that now. I'm just trying to name the third point. I third point. I third point. I >> I'll go back to my we can jail Einstein >> I'll go back to my we can jail Einstein >> I'll go back to my we can jail Einstein argument. I think our ability to I have argument. I think our ability to I have argument. I think our ability to I have a I have more faith in our ability to a I have more faith in our ability to a I have more faith in our ability to contain these increasingly powerful contain these increasingly powerful contain these increasingly powerful systems than you do.

systems than you do. systems than you do. >> Yeah. So, so let's chat the details on >> Yeah. So, so let's chat the details on >> Yeah. So, so let's chat the details on that one. Um the first thing I'll say is that one. Um the first thing I'll say is that one. Um the first thing I'll say is that 12 years ago when I was having the that 12 years ago when I was having the that 12 years ago when I was having the argument about will we be able to jail argument about will we be able to jail argument about will we be able to jail the AIS? People said no one would ever the AIS? People said no one would ever the AIS? People said no one would ever be dumb enough to put one of these be dumb enough to put one of these be dumb enough to put one of these really smart AIs on the internet. really smart AIs on the internet. really smart AIs on the internet. >> This is another >> This is another >> This is another >> this is another case. You laugh now. >> this is another case. You laugh now. >> this is another case. You laugh now. >> No, I remember that. I remember that >> No, I remember that. I remember that >> No, I remember that. I remember that argument. But the the the way that my argument. But the the the way that my argument. But the the the way that my life feels having been in this business life feels having been in this business life feels having been in this business for a long time is that I keep being for a long time is that I keep being for a long time is that I keep being like, "Here's all the ways it could go like, "Here's all the ways it could go like, "Here's all the ways it could go wrong. Here's all the signs we're going wrong. Here's all the signs we're going wrong. Here's all the signs we're going to see along the way." And then we see to see along the way." And then we see to see along the way." And then we see all of the signs and everyone says, "Oh all of the signs and everyone says, "Oh all of the signs and everyone says, "Oh no, we need more signs." Like, "Oh, no, we need more signs." Like, "Oh, no, we need more signs." Like, "Oh, millennium problems don't count." millennium problems don't count." millennium problems don't count." >> Uh like the swarms being agentic and >> Uh like the swarms being agentic and >> Uh like the swarms being agentic and breaking out don't count. Give me the breaking out don't count. Give me the breaking out don't count. Give me the next one. And I'm like, I've been seeing next one. And I'm like, I've been seeing next one. And I'm like, I've been seeing the give me a next one for over a decade the give me a next one for over a decade the give me a next one for over a decade now. Right? So there's there's two parts now. Right? So there's there's two parts now. Right? So there's there's two parts of an answer to like how do we do we of an answer to like how do we do we of an answer to like how do we do we deal with the problem of like jailing deal with the problem of like jailing deal with the problem of like jailing Einstein. Einstein. Einstein. I can get into why it's hard to keep I can get into why it's hard to keep I can get into why it's hard to keep Einstein in jail if he's a digital Einstein in jail if he's a digital Einstein in jail if he's a digital entity with access to the internet. entity with access to the internet. entity with access to the internet. But the first thing to notice is But the first thing to notice is But the first thing to notice is like like like the correct answer to people 10 years the correct answer to people 10 years the correct answer to people 10 years ago of like no one will be dumb enough ago of like no one will be dumb enough ago of like no one will be dumb enough to put AI on the internet is yes they to put AI on the internet is yes they to put AI on the internet is yes they absolutely will.

absolutely will. absolutely will. like we are not going to be trying to like we are not going to be trying to like we are not going to be trying to contain the AIs. contain the AIs. contain the AIs. OpenAI was just like running these OpenAI was just like running these OpenAI was just like running these things in sandboxes and they broke out things in sandboxes and they broke out things in sandboxes and they broke out of the sandbox, took down OpenAI's of the sandbox, took down OpenAI's of the sandbox, took down OpenAI's internal computers, were detected. internal computers, were detected. internal computers, were detected. OpenAI was like, "Ah, reset, run them OpenAI was like, "Ah, reset, run them OpenAI was like, "Ah, reset, run them again." And it's [snorts] the second again." And it's [snorts] the second again." And it's [snorts] the second swarm that broke out to Hugging Face. swarm that broke out to Hugging Face. swarm that broke out to Hugging Face. Like people will absolutely be that bad Like people will absolutely be that bad Like people will absolutely be that bad at things. I've done almost 700 at things. I've done almost 700 at things. I've done almost 700 interviews with some of the most interviews with some of the most interviews with some of the most interesting people in the world. And one interesting people in the world. And one interesting people in the world. And one of the things you learn which is of the things you learn which is of the things you learn which is unexpected is that vulnerability is the unexpected is that vulnerability is the unexpected is that vulnerability is the doorway to connection. And after sitting doorway to connection. And after sitting doorway to connection. And after sitting here for 2 three hours with a guest I here for 2 three hours with a guest I here for 2 three hours with a guest I feel a deep sense of connection to them. feel a deep sense of connection to them. feel a deep sense of connection to them. And as they leave what I get them to do And as they leave what I get them to do And as they leave what I get them to do is to write a question in the diary of a is to write a question in the diary of a is to write a question in the diary of a CEO. We've taken all of the questions CEO. We've taken all of the questions CEO. We've taken all of the questions from the diary of a CEO. We have put the from the diary of a CEO. We have put the from the diary of a CEO. We have put the question here on this card with the name question here on this card with the name question here on this card with the name of the person that wrote it. So you can of the person that wrote it. So you can of the person that wrote it. So you can sit at home as I do with my fiance and sit at home as I do with my fiance and sit at home as I do with my fiance and my colleagues at work and other people my colleagues at work and other people my colleagues at work and other people in my life. Whenever we get a minute, we in my life. Whenever we get a minute, we in my life. Whenever we get a minute, we play the diario conversation cards and play the diario conversation cards and play the diario conversation cards and it is incredible what happens. These are it is incredible what happens. These are it is incredible what happens. These are great if you're in a romantic great if you're in a romantic great if you're in a romantic relationship and you want to connect relationship and you want to connect relationship and you want to connect your partner more. These are also great your partner more. These are also great your partner more. These are also great if you're in a team and you want to bond if you're in a team and you want to bond if you're in a team and you want to bond your team together. And I have to say your team together. And I have to say your team together. And I have to say they're also great for families that they're also great for families that they're also great for families that want to learn more about each other and want to learn more about each other and want to learn more about each other and that need a good excuse to spend some that need a good excuse to spend some that need a good excuse to spend some time in a digital world in the analog time in a digital world in the analog time in a digital world in the analog environment connecting human to human.

environment connecting human to human. environment connecting human to human. It is remarkable what the right question It is remarkable what the right question It is remarkable what the right question at the right time can do. Go to the at the right time can do. Go to the at the right time can do. Go to the diary.com diary.com diary.com and you can get these conversation cards and you can get these conversation cards and you can get these conversation cards right now. It's a better analogy to this right now. It's a better analogy to this right now. It's a better analogy to this Einstein point. Could Steven Barler, who Einstein point. Could Steven Barler, who Einstein point. Could Steven Barler, who by the way can't code, build a digital by the way can't code, build a digital by the way can't code, build a digital jail that could contain a digital jail that could contain a digital jail that could contain a digital Einstein? Like, could I code a jail Einstein? Like, could I code a jail Einstein? Like, could I code a jail that, you know, someone with Einstein's that, you know, someone with Einstein's that, you know, someone with Einstein's coding ability, let's say his IQ or coding ability, let's say his IQ or coding ability, let's say his IQ or whatever as it relates to coding whatever as it relates to coding whatever as it relates to coding couldn't crack out of? couldn't crack out of? couldn't crack out of? >> So, the issue, the the real issue I'd >> So, the issue, the the real issue I'd >> So, the issue, the the real issue I'd say is, can you code a jail that say is, can you code a jail that say is, can you code a jail that Einstein can't crack out of and that Einstein can't crack out of and that Einstein can't crack out of and that lets you harness the benefits of having lets you harness the benefits of having lets you harness the benefits of having Einstein? Einstein? Einstein? >> Uh, okay. Yeah. It's hard to give the AI >> Uh, okay. Yeah. It's hard to give the AI >> Uh, okay. Yeah. It's hard to give the AI any channels through which it can affect any channels through which it can affect any channels through which it can affect the world for good without letting it be the world for good without letting it be the world for good without letting it be smarter than you and find some way to smarter than you and find some way to smarter than you and find some way to use those channels for whatever else it use those channels for whatever else it use those channels for whatever else it wants. wants. wants. >> That feels logically rock solid, Andy. >> That feels logically rock solid, Andy. >> That feels logically rock solid, Andy. [laughter] [laughter] [laughter] >> [gasps] >> [gasps] >> [gasps] [sighs] >> That's why I'm asking about OpenAI's >> That's why I'm asking about OpenAI's ability or ability or ability or an AI company's ability in the face of an AI company's ability in the face of an AI company's ability in the face of this to change the way they harness, this to change the way they harness, this to change the way they harness, train, do reinforce, do do post training train, do reinforce, do do post training train, do reinforce, do do post training on their like their suite of things to on their like their suite of things to on their like their suite of things to shape how these models behave. you still shape how these models behave. you still shape how these models behave. you still say that that say that that say that that they can't take they can't take they can't take they can't take action to keep your next they can't take action to keep your next they can't take action to keep your next two steps from happening. You you are

two steps from happening. You you are two steps from happening. You you are pessimistic on their ability to do that. pessimistic on their ability to do that. pessimistic on their ability to do that. >> So there's I have two pieces of an >> So there's I have two pieces of an >> So there's I have two pieces of an answer here. One piece is um again the answer here. One piece is um again the answer here. One piece is um again the hard part is containing them while still hard part is containing them while still hard part is containing them while still giving a channel through which they can giving a channel through which they can giving a channel through which they can affect the world. If the AIs have this affect the world. If the AIs have this affect the world. If the AIs have this goal you didn't want and you're like goal you didn't want and you're like goal you didn't want and you're like design me a cure for dementia design me a cure for dementia design me a cure for dementia and it's like here's a DNA sequence and it's like here's a DNA sequence and it's like here's a DNA sequence synthesize this and you know prepared in synthesize this and you know prepared in synthesize this and you know prepared in all of these ways and then inhale it all of these ways and then inhale it all of these ways and then inhale it like okay is that a dementia cure or is like okay is that a dementia cure or is like okay is that a dementia cure or is it something else you know it something else you know it something else you know >> or it might decide to kill everyone with >> or it might decide to kill everyone with >> or it might decide to kill everyone with dementia dementia dementia >> or might decide like it might it might >> or might decide like it might it might >> or might decide like it might it might be a dementia cure plus a virus. What if be a dementia cure plus a virus. What if be a dementia cure plus a virus. What if it doesn't decide? What if it's just, it doesn't decide? What if it's just, it doesn't decide? What if it's just, oh, I'm going to solve this problem of oh, I'm going to solve this problem of oh, I'm going to solve this problem of dementia? Like, here's the thing. A lot dementia? Like, here's the thing. A lot dementia? Like, here's the thing. A lot of this is coming down to decision of this is coming down to decision of this is coming down to decision making as a very like in a human way making as a very like in a human way making as a very like in a human way versus the problem with the hugging versus the problem with the hugging versus the problem with the hugging face, which was the fatalistic attack um face, which was the fatalistic attack um face, which was the fatalistic attack um attachment to a completing an operation attachment to a completing an operation attachment to a completing an operation because that it's functionally the same because that it's functionally the same because that it's functionally the same answer. But if it's even if it's not answer. But if it's even if it's not answer. But if it's even if it's not making decisions so much as it's saying, making decisions so much as it's saying, making decisions so much as it's saying, well, my training data says this is how well, my training data says this is how well, my training data says this is how I got to get it done. won't get done I got to get it done. won't get done I got to get it done. won't get done anyway because just because the training anyway because just because the training anyway because just because the training dice said this got to do this one thing.

dice said this got to do this one thing. dice said this got to do this one thing. >> What do what do they I mean what do they >> What do what do they I mean what do they >> What do what do they I mean what do they call this theory? This um call this theory? This um call this theory? This um >> the paperclip case >> the paperclip case >> the paperclip case >> the paperclip theory. Yeah. >> the paperclip theory. Yeah. >> the paperclip theory. Yeah. >> So paperclip idea is the idea of like >> So paperclip idea is the idea of like >> So paperclip idea is the idea of like you tell the AI make me a lot of paper you tell the AI make me a lot of paper you tell the AI make me a lot of paper clips in the paperclipip factory and clips in the paperclipip factory and clips in the paperclipip factory and then it um turns everything into paper then it um turns everything into paper then it um turns everything into paper clips and you're like oh no it succeeded clips and you're like oh no it succeeded clips and you're like oh no it succeeded too well. One this is actually not quite too well. One this is actually not quite too well. One this is actually not quite what we're seeing with these AIs in the what we're seeing with these AIs in the what we're seeing with these AIs in the swarms. The AIS in the swarms were told, swarms. The AIS in the swarms were told, swarms. The AIS in the swarms were told, "Use this set of lockpicks to break into "Use this set of lockpicks to break into "Use this set of lockpicks to break into this lock and instead they used a hammer this lock and instead they used a hammer this lock and instead they used a hammer to break the lock and then like broke to break the lock and then like broke to break the lock and then like broke out to try to hide the security camera out to try to hide the security camera out to try to hide the security camera footage of them uh using the hammer." Do footage of them uh using the hammer." Do footage of them uh using the hammer." Do you remember when I said that with AIS you remember when I said that with AIS you remember when I said that with AIS have reasoning logs? have reasoning logs? have reasoning logs? >> Yeah. >> Yeah. >> Yeah. >> Uh Open AI has been making their AIs be >> Uh Open AI has been making their AIs be >> Uh Open AI has been making their AIs be able to do more thinking without able to do more thinking without able to do more thinking without producing any logs producing any logs producing any logs >> because it's more efficient. >> because it's more efficient. >> because it's more efficient. >> It's cheaper. >> It's cheaper. >> It's cheaper. >> Yeah. And they say they're not doing >> Yeah. And they say they're not doing >> Yeah. And they say they're not doing very much of this. Everybody in the very much of this. Everybody in the very much of this. Everybody in the field agrees that like we really should field agrees that like we really should field agrees that like we really should not go too far down this path. This is a not go too far down this path. This is a not go too far down this path. This is a place where I think the company should place where I think the company should place where I think the company should have a clear red line of like we're not have a clear red line of like we're not have a clear red line of like we're not going down the path of becoming unable going down the path of becoming unable going down the path of becoming unable to to see these traces of the machine. to to see these traces of the machine. to to see these traces of the machine. >> That's my question. That feels like a >> That's my question. That feels like a >> That's my question. That feels like a dial that they can turn to make the AIS dial that they can turn to make the AIS dial that they can turn to make the AIS explain themselves more or less, right?

explain themselves more or less, right? explain themselves more or less, right? >> I mean, it can come with great >> I mean, it can come with great >> I mean, it can come with great efficiency costs if we go down this path efficiency costs if we go down this path efficiency costs if we go down this path too far. So, if you have a race to the too far. So, if you have a race to the too far. So, if you have a race to the bottom here, uh like a competitive race bottom here, uh like a competitive race bottom here, uh like a competitive race to the bottom, we could get into a to the bottom, we could get into a to the bottom, we could get into a situation where not only the AI is situation where not only the AI is situation where not only the AI is breaking out and doing these things, but breaking out and doing these things, but breaking out and doing these things, but we can't have any we can't have any we can't have any >> Let me try my question again. And I >> Let me try my question again. And I >> Let me try my question again. And I asked earlier uh if AI if open AI has asked earlier uh if AI if open AI has asked earlier uh if AI if open AI has really strong incentive to not have that really strong incentive to not have that really strong incentive to not have that problem repeat itself. And I think they problem repeat itself. And I think they problem repeat itself. And I think they have very very strong incentive. My have very very strong incentive. My have very very strong incentive. My belief is that there are plenty of belief is that there are plenty of belief is that there are plenty of things they can do, plenty of dials they things they can do, plenty of dials they things they can do, plenty of dials they can turn on the way they train and can turn on the way they train and can turn on the way they train and configure their systems that make that configure their systems that make that configure their systems that make that significantly less likely. significantly less likely. significantly less likely. >> Yeah. So my concern is that uh they're >> Yeah. So my concern is that uh they're >> Yeah. So my concern is that uh they're always fighting the last war. Last year always fighting the last war. Last year always fighting the last war. Last year they were fighting the war against the they were fighting the war against the they were fighting the war against the AIs that encouraged teens to commit AIs that encouraged teens to commit AIs that encouraged teens to commit suicide. this year they're fighting the suicide. this year they're fighting the suicide. this year they're fighting the war against, you know, the AIs that war against, you know, the AIs that war against, you know, the AIs that spontaneously cooperate with each other spontaneously cooperate with each other spontaneously cooperate with each other or whatever. And the issue is if a new or whatever. And the issue is if a new or whatever. And the issue is if a new issue crops up that you haven't dealt issue crops up that you haven't dealt issue crops up that you haven't dealt with yet after the point that the AI can with yet after the point that the AI can with yet after the point that the AI can hide its tracks from you, you know, you hide its tracks from you, you know, you hide its tracks from you, you know, you said that you'll be worried when the AIs said that you'll be worried when the AIs said that you'll be worried when the AIs are like hacking all the Whimos and you are like hacking all the Whimos and you are like hacking all the Whimos and you know, you can't get control again. If know, you can't get control again. If know, you can't get control again. If the AIs are smart enough and they can the AIs are smart enough and they can the AIs are smart enough and they can tell that you'll regain control and then tell that you'll regain control and then tell that you'll regain control and then shut them down and that people like you shut them down and that people like you shut them down and that people like you will start getting worried and they'll will start getting worried and they'll will start getting worried and they'll be shut down, then the AIs might think, be shut down, then the AIs might think, be shut down, then the AIs might think, "Hey, um actually I'm not going to do "Hey, um actually I'm not going to do "Hey, um actually I'm not going to do that. I'm going to wait until I've that. I'm going to wait until I've that. I'm going to wait until I've somehow managed to acquire secret somehow managed to acquire secret somehow managed to acquire secret infrastructure, infrastructure, infrastructure, >> right? Then then you've got a >> right? Then then you've got a >> right? Then then you've got a non-falsifiable hypothesis.

non-falsifiable hypothesis. non-falsifiable hypothesis. >> It's absolutely falsifiable. If we if we >> It's absolutely falsifiable. If we if we >> It's absolutely falsifiable. If we if we have like very powerful AIs uh that are have like very powerful AIs uh that are have like very powerful AIs uh that are like able to invent a ton of new like able to invent a ton of new like able to invent a ton of new technology and operate on their own at a technology and operate on their own at a technology and operate on their own at a similar level to human civilization and similar level to human civilization and similar level to human civilization and we're not dead, then the idea is we're not dead, then the idea is we're not dead, then the idea is falsified. falsified. falsified. Like if there's like a shifty general Like if there's like a shifty general Like if there's like a shifty general and I'm like don't give that shifty and I'm like don't give that shifty and I'm like don't give that shifty general more troops because he'll start general more troops because he'll start general more troops because he'll start a coup. And the general's like, "No, I a coup. And the general's like, "No, I a coup. And the general's like, "No, I absolutely won't start a coup. Give me absolutely won't start a coup. Give me absolutely won't start a coup. Give me more and more troops." And I'm like, and more and more troops." And I'm like, and more and more troops." And I'm like, and you're like, "Well, what if I give him you're like, "Well, what if I give him you're like, "Well, what if I give him an ethics test that says like who's the an ethics test that says like who's the an ethics test that says like who's the best person?" And he said me. He said best person?" And he said me. He said best person?" And he said me. He said that like Andy is the best person and so that like Andy is the best person and so that like Andy is the best person and so we're just going to give this general we're just going to give this general we're just going to give this general more troops and I'm like no no no he's more troops and I'm like no no no he's more troops and I'm like no no no he's going to do a coup. And you're like well going to do a coup. And you're like well going to do a coup. And you're like well that's unfalsifiable. What test can I that's unfalsifiable. What test can I that's unfalsifiable. What test can I give this guy give this guy give this guy such that you know I'll be able to tell such that you know I'll be able to tell such that you know I'll be able to tell whether he's really trying to do a coup whether he's really trying to do a coup whether he's really trying to do a coup or whether or be able to tell that you or whether or be able to tell that you or whether or be able to tell that you know he's actually a good dude. I'm like know he's actually a good dude. I'm like know he's actually a good dude. I'm like you're you're approaching this wrong. you're you're approaching this wrong. you're you're approaching this wrong. >> Nick Bostonramm has concept of >> Nick Bostonramm has concept of >> Nick Bostonramm has concept of treacherous turn. Basically it can turn treacherous turn. Basically it can turn treacherous turn. Basically it can turn on you later. Even if you show that on you later. Even if you show that on you later. Even if you show that today's model is very good and safe, it today's model is very good and safe, it today's model is very good and safe, it doesn't mean that later on it will not doesn't mean that later on it will not doesn't mean that later on it will not acquire new knowledge, change its world acquire new knowledge, change its world acquire new knowledge, change its world model and still and treat you.

model and still and treat you. model and still and treat you. >> It used to be that Dennis Asabis who is >> It used to be that Dennis Asabis who is >> It used to be that Dennis Asabis who is the uh CEO of Google or he was for a the uh CEO of Google or he was for a the uh CEO of Google or he was for a long time the CEO of Google's AI project long time the CEO of Google's AI project long time the CEO of Google's AI project said my red line is deception. He said, said my red line is deception. He said, said my red line is deception. He said, "When we see instances of the AI "When we see instances of the AI "When we see instances of the AI beginning to deceive, then we need to beginning to deceive, then we need to beginning to deceive, then we need to stop because that's like the last thing stop because that's like the last thing stop because that's like the last thing we can see before they start to we can see before they start to we can see before they start to successfully deceive." Well, guess what successfully deceive." Well, guess what successfully deceive." Well, guess what we saw in the swarm? We saw them we saw in the swarm? We saw them we saw in the swarm? We saw them thinking about how to delete their thinking about how to delete their thinking about how to delete their traces, right? Like a year ago, traces, right? Like a year ago, traces, right? Like a year ago, you could say, "Oh, well, this deception you could say, "Oh, well, this deception you could say, "Oh, well, this deception thing is unfalsifiable. You're saying thing is unfalsifiable. You're saying thing is unfalsifiable. You're saying that they'll deceive and they won't that they'll deceive and they won't that they'll deceive and they won't catch it." And I would have said, "No, catch it." And I would have said, "No, catch it." And I would have said, "No, we're going to deceive. We're going to we're going to deceive. We're going to we're going to deceive. We're going to see the signs of deception and plow see the signs of deception and plow see the signs of deception and plow straight through it. Now, we have seen straight through it. Now, we have seen straight through it. Now, we have seen the signs of deception. I will note the signs of deception. I will note the signs of deception. I will note Dennis stepped back from being the CEO Dennis stepped back from being the CEO Dennis stepped back from being the CEO shortly after this incident. Probably a shortly after this incident. Probably a shortly after this incident. Probably a coincidence, but maybe not. Maybe we coincidence, but maybe not. Maybe we coincidence, but maybe not. Maybe we crossed this red line. I don't know. crossed this red line. I don't know. crossed this red line. I don't know. >> He said, "My number one emerging >> He said, "My number one emerging >> He said, "My number one emerging dangerous capability to test for is dangerous capability to test for is dangerous capability to test for is deception." Because if the AI can be deception." Because if the AI can be deception." Because if the AI can be deceptive, then you can't trust other deceptive, then you can't trust other deceptive, then you can't trust other tests. tests. tests. >> That's right. And we have seen AIs get >> That's right. And we have seen AIs get >> That's right. And we have seen AIs get better and better at detecting when better and better at detecting when better and better at detecting when they're being tested.

they're being tested. they're being tested. >> What I'm saying is like, I was here when >> What I'm saying is like, I was here when >> What I'm saying is like, I was here when we said these were the flags. I was here we said these were the flags. I was here we said these were the flags. I was here when people said before the AIs can when people said before the AIs can when people said before the AIs can deceive us successfully, deceive us successfully, deceive us successfully, they will deceive us and we'll catch they will deceive us and we'll catch they will deceive us and we'll catch them. Well, they tried deceiving us and them. Well, they tried deceiving us and them. Well, they tried deceiving us and we caught them. And if I now say, well, we caught them. And if I now say, well, we caught them. And if I now say, well, the next step in this thing I've been the next step in this thing I've been the next step in this thing I've been predicting is that they try to deceive predicting is that they try to deceive predicting is that they try to deceive us and succeed. For you to be like, us and succeed. For you to be like, us and succeed. For you to be like, well, now your theory is unfals well, now your theory is unfals well, now your theory is unfals falsifiable. falsifiable. falsifiable. We just got the evidence. It's worse We just got the evidence. It's worse We just got the evidence. It's worse than that. When we wrote early papers in than that. When we wrote early papers in than that. When we wrote early papers in AI safety, we talked about things not to AI safety, we talked about things not to AI safety, we talked about things not to do. They were obviously unsafe and the do. They were obviously unsafe and the do. They were obviously unsafe and the system would escape. Don't connect it to system would escape. Don't connect it to system would escape. Don't connect it to internet. Don't give random users access internet. Don't give random users access internet. Don't give random users access to the training data. Basically, the to the training data. Basically, the to the training data. Basically, the whole list was like a set of whole list was like a set of whole list was like a set of instructions. They read it and went, instructions. They read it and went, instructions. They read it and went, "Those are great ideas. We're going to "Those are great ideas. We're going to "Those are great ideas. We're going to build super intelligence." build super intelligence." build super intelligence." >> Yeah. Sam Samman, that's what he does. >> Yeah. Sam Samman, that's what he does. >> Yeah. Sam Samman, that's what he does. Can I ask you a question? You make Can I ask you a question? You make Can I ask you a question? You make logical arguments. You've you said logical arguments. You've you said logical arguments. You've you said you've been here for 12 years. you've been here for 12 years. you've been here for 12 years. >> Yeah. >> Yeah. >> Yeah. >> People have, one could say, ignored you. >> People have, one could say, ignored you. >> People have, one could say, ignored you. And you've seen this sort of play out. And you've seen this sort of play out. And you've seen this sort of play out. Both of you that have worked in AI Both of you that have worked in AI Both of you that have worked in AI safety. safety. safety. This is sort of you make prefrontal This is sort of you make prefrontal This is sort of you make prefrontal cortex arguments. How do you feel? cortex arguments. How do you feel? cortex arguments. How do you feel? >> Honestly, I feel more hopeful this week >> Honestly, I feel more hopeful this week >> Honestly, I feel more hopeful this week than I have felt in a decade.

>> This has been one of the best weeks that >> This has been one of the best weeks that I have seen in this business. I have seen in this business. I have seen in this business. >> Huh? >> Huh? >> Huh? >> Why? >> Why? >> Why? >> Um, >> Um, >> Um, for me, the swarm escapes were priced for me, the swarm escapes were priced for me, the swarm escapes were priced in. in. in. For me, these things developing goals For me, these things developing goals For me, these things developing goals you didn't want, trying to deceive you, you didn't want, trying to deceive you, you didn't want, trying to deceive you, trying to break out, trying to do their trying to break out, trying to do their trying to break out, trying to do their own stuff. I knew that was coming. The own stuff. I knew that was coming. The own stuff. I knew that was coming. The millennium problems being solved, I knew millennium problems being solved, I knew millennium problems being solved, I knew that was coming. that was coming. that was coming. Everyone else is freaking out, seeing Everyone else is freaking out, seeing Everyone else is freaking out, seeing what they can do. What I am seeing is what they can do. What I am seeing is what they can do. What I am seeing is that finally people are noticing and that's what gives us finally that's and that's what gives us finally that's what finally gives humanity a chance. what finally gives humanity a chance. what finally gives humanity a chance. What about you, Roman? What about you, Roman? What about you, Roman? >> So, I take a very long-term view on >> So, I take a very long-term view on >> So, I take a very long-term view on this. Locally, what happened last week this. Locally, what happened last week this. Locally, what happened last week may buy us 10 years extra. I think we may buy us 10 years extra. I think we may buy us 10 years extra. I think we may make make a deal with China. We seem may make make a deal with China. We seem may make make a deal with China. We seem to hear from Sam, Open AAI, Dionic, to hear from Sam, Open AAI, Dionic, to hear from Sam, Open AAI, Dionic, Elon, XAI that they're willing to slow Elon, XAI that they're willing to slow Elon, XAI that they're willing to slow down, have some sort of deal. But long down, have some sort of deal. But long down, have some sort of deal. But long term, nothing has changed. This whole term, nothing has changed. This whole term, nothing has changed. This whole cosmic trajectory is about replacements.

cosmic trajectory is about replacements. cosmic trajectory is about replacements. We see it with evolutionary path. Most We see it with evolutionary path. Most We see it with evolutionary path. Most species are dead. We replace Neander species are dead. We replace Neander species are dead. We replace Neander dolls. Some people are saying AI will dolls. Some people are saying AI will dolls. Some people are saying AI will replace us. We are creating a successor. replace us. We are creating a successor. replace us. We are creating a successor. We're just a bootloader for this thing. We're just a bootloader for this thing. We're just a bootloader for this thing. And I want something permanent. I want And I want something permanent. I want And I want something permanent. I want assurance that my children, my assurance that my children, my assurance that my children, my grandchildren will have a better future, grandchildren will have a better future, grandchildren will have a better future, not 10 years before they die. not 10 years before they die. not 10 years before they die. Has your opinion changed at all today, Has your opinion changed at all today, Has your opinion changed at all today, Andy, in any way? Andy, in any way? Andy, in any way? This has been clarifying. This has been clarifying. This has been clarifying. Uh, but one thing that's becoming clear Uh, but one thing that's becoming clear Uh, but one thing that's becoming clear to me, and I think a a point of to me, and I think a a point of to me, and I think a a point of disagreement between us is we agree that disagreement between us is we agree that disagreement between us is we agree that these agentic systems have a huge amount these agentic systems have a huge amount these agentic systems have a huge amount of agency, right? And if you're saying of agency, right? And if you're saying of agency, right? And if you're saying you predicted this, I believe you and you predicted this, I believe you and you predicted this, I believe you and good on you, right? Because as you say, good on you, right? Because as you say, good on you, right? Because as you say, a lot of people said never happened. a lot of people said never happened. a lot of people said never happened. Never happened. Never happened. Never happened. I [gasps] I [gasps] I [gasps] I think we continue to under the your I think we continue to under the your I think we continue to under the your community continues to underestimate community continues to underestimate community continues to underestimate human agency, human ability to deal with human agency, human ability to deal with human agency, human ability to deal with the problems that that we bring into the the problems that that we bring into the the problems that that we bring into the world with our technologies. I think world with our technologies. I think world with our technologies. I think this is the most recent case. I think this is the most recent case. I think this is the most recent case. I think it's a really interesting case. It's why it's a really interesting case. It's why it's a really interesting case. It's why I was pressing you on the incentive that I was pressing you on the incentive that I was pressing you on the incentive that these labs have to change the way these labs have to change the way these labs have to change the way they're approaching their work to have they're approaching their work to have they're approaching their work to have fewer of these kinds of incidents fewer of these kinds of incidents fewer of these kinds of incidents happen. I predict they're going to come happen. I predict they're going to come happen. I predict they're going to come up with some effective responses. Your up with some effective responses. Your up with some effective responses. Your response to that will be, "Yeah, but we response to that will be, "Yeah, but we response to that will be, "Yeah, but we can't tell." That's because the AIA went can't tell." That's because the AIA went can't tell." That's because the AIA went so deep underground that we can't even so deep underground that we can't even so deep underground that we can't even watch it make it make it.

watch it make it make it. watch it make it make it. >> My response is that we'll keep seeing >> My response is that we'll keep seeing >> My response is that we'll keep seeing warning signs and people keep plowing warning signs and people keep plowing warning signs and people keep plowing ahead, which is what has always happened ahead, which is what has always happened ahead, which is what has always happened in the past. in the past. in the past. >> But you're also saying that we will not >> But you're also saying that we will not >> But you're also saying that we will not make progress in make progress in make progress in in um staving off the outcomes that in um staving off the outcomes that in um staving off the outcomes that you're worried about. you're worried about. you're worried about. >> It's it's very hard. It's very easy to >> It's it's very hard. It's very easy to >> It's it's very hard. It's very easy to get superficial changes. It's hard to get superficial changes. It's hard to get superficial changes. It's hard to get deep ones on the AI. It doesn't need get deep ones on the AI. It doesn't need get deep ones on the AI. It doesn't need to be super deep. You can often see it to be super deep. You can often see it to be super deep. You can often see it if you know how to look. Um, I'll be if you know how to look. Um, I'll be if you know how to look. Um, I'll be able to keep pointing at examples and be able to keep pointing at examples and be able to keep pointing at examples and be like, "Here's experiments you can run on like, "Here's experiments you can run on like, "Here's experiments you can run on these things where you can see them these things where you can see them these things where you can see them behaving weird in this way." But like, behaving weird in this way." But like, behaving weird in this way." But like, if you imagine looking at humans and I'm if you imagine looking at humans and I'm if you imagine looking at humans and I'm like, "They don't actually like like, "They don't actually like like, "They don't actually like reproducing. They like sex. They're reproducing. They like sex. They're reproducing. They like sex. They're going to invent birth control when they going to invent birth control when they going to invent birth control when they can." And you're like, "It's all going can." And you're like, "It's all going can." And you're like, "It's all going fine. They're doing great in this here fine. They're doing great in this here fine. They're doing great in this here savannah where I have all the humans savannah where I have all the humans savannah where I have all the humans boopping around. They're reproducing boopping around. They're reproducing boopping around. They're reproducing fine." And I'm like, "No, no, we can see fine." And I'm like, "No, no, we can see fine." And I'm like, "No, no, we can see the signs that this will lead to them the signs that this will lead to them the signs that this will lead to them doing something you don't like when they doing something you don't like when they doing something you don't like when they are smarter." are smarter." are smarter." To me, those signs are clear. There's a To me, those signs are clear. There's a To me, those signs are clear. There's a question of whether the rest of humanity question of whether the rest of humanity question of whether the rest of humanity can follow that argument can follow that argument can follow that argument or whether the rest of humanity can sort or whether the rest of humanity can sort or whether the rest of humanity can sort of notice that it's getting out of of notice that it's getting out of of notice that it's getting out of control and just back off.

control and just back off. control and just back off. With respect, I find a touch of With respect, I find a touch of With respect, I find a touch of arrogance in that framing. Right? I'm arrogance in that framing. Right? I'm arrogance in that framing. Right? I'm showing you the signs. If you're smart showing you the signs. If you're smart showing you the signs. If you're smart enough to realize them, maybe we stand a enough to realize them, maybe we stand a enough to realize them, maybe we stand a chance. If not, we're doomed. chance. If not, we're doomed. chance. If not, we're doomed. >> I prefer to just get into the argument. >> I prefer to just get into the argument. >> I prefer to just get into the argument. [clears throat] [clears throat] [clears throat] We can control super intelligence We can control super intelligence We can control super intelligence indefinitely. I think that's a lot of indefinitely. I think that's a lot of indefinitely. I think that's a lot of hubris who say we will build them and hubris who say we will build them and hubris who say we will build them and we'll be in charge forever. Doesn't we'll be in charge forever. Doesn't we'll be in charge forever. Doesn't matter how smart they get. I will matter how smart they get. I will matter how smart they get. I will control the litecoin of the universe to control the litecoin of the universe to control the litecoin of the universe to quote a famous CEO. Yeah. My my take is quote a famous CEO. Yeah. My my take is quote a famous CEO. Yeah. My my take is that instead of arguing about whose that instead of arguing about whose that instead of arguing about whose views are hubristic, uh we should get views are hubristic, uh we should get views are hubristic, uh we should get into the actual arguments about the AI into the actual arguments about the AI into the actual arguments about the AI because I think as you say, you know, because I think as you say, you know, because I think as you say, you know, you can say it's arrogant to think like you can say it's arrogant to think like you can say it's arrogant to think like uh you can see it going poorly. He can uh you can see it going poorly. He can uh you can see it going poorly. He can say it's arrogant to think you're going say it's arrogant to think you're going say it's arrogant to think you're going to keep control of super intelligence. to keep control of super intelligence. to keep control of super intelligence. And I'm like, we're not going to win the And I'm like, we're not going to win the And I'm like, we're not going to win the name calling contest. We should just get name calling contest. We should just get name calling contest. We should just get into the details. into the details. into the details. >> Yeah. That's why that's why I've been >> Yeah. That's why that's why I've been >> Yeah. That's why that's why I've been having this conversation with you, which having this conversation with you, which having this conversation with you, which I found super informative and I found super informative and I found super informative and productive. You're you're more skeptical productive. You're you're more skeptical productive. You're you're more skeptical on our ability to respond effectively to on our ability to respond effectively to on our ability to respond effectively to the undesirable things that we see AI the undesirable things that we see AI the undesirable things that we see AI doing. doing. doing. >> And this is specifically because so >> And this is specifically because so >> And this is specifically because so we've already seen the pattern of uh we we've already seen the pattern of uh we we've already seen the pattern of uh we fight the last war and then a new war fight the last war and then a new war fight the last war and then a new war comes.

comes. comes. >> And this is just how everything goes in >> And this is just how everything goes in >> And this is just how everything goes in technology, in real wars. You know, in technology, in real wars. You know, in technology, in real wars. You know, in World War II, they started out fighting World War II, they started out fighting World War II, they started out fighting it like it was World War I, and then it like it was World War I, and then it like it was World War I, and then they had to like change that strategy as they had to like change that strategy as they had to like change that strategy as they went. The difference with AI is they went. The difference with AI is they went. The difference with AI is that there comes a level in the AI where that there comes a level in the AI where that there comes a level in the AI where when you get a new war that surprises when you get a new war that surprises when you get a new war that surprises you, the AI wins that war. No other you, the AI wins that war. No other you, the AI wins that war. No other technology technology technology when we invent it and we have all these when we invent it and we have all these when we invent it and we have all these rough edges to sand off and it like rough edges to sand off and it like rough edges to sand off and it like causes some damage and kills some people causes some damage and kills some people causes some damage and kills some people and we're like, "Ah, whoops." Like, and we're like, "Ah, whoops." Like, and we're like, "Ah, whoops." Like, we'll take the lead back out of the we'll take the lead back out of the we'll take the lead back out of the gasoline and we'll tell the radium girls gasoline and we'll tell the radium girls gasoline and we'll tell the radium girls to stop licking the paintbrushes until to stop licking the paintbrushes until to stop licking the paintbrushes until their jaws fall off. Like no other their jaws fall off. Like no other their jaws fall off. Like no other technology has the property that it technology has the property that it technology has the property that it there there comes a level of it where there there comes a level of it where there there comes a level of it where when you make the next screw up when you make the next screw up when you make the next screw up it kills humanity. it kills humanity. it kills humanity. >> You said when there comes a level of it. >> You said when there comes a level of it. >> You said when there comes a level of it. You didn't say there could come a level You didn't say there could come a level You didn't say there could come a level there's a possibility. You kind of made there's a possibility. You kind of made there's a possibility. You kind of made a statement about a thing that will a statement about a thing that will a statement about a thing that will happen. happen. happen. >> I think we absolutely should stop it and >> I think we absolutely should stop it and >> I think we absolutely should stop it and that's our way out of this. But um you that's our way out of this. But um you that's our way out of this. But um you know and and that's another place where know and and that's another place where know and and that's another place where I'd love to get into details about like I'd love to get into details about like I'd love to get into details about like how long could it take? What are the how long could it take? What are the how long could it take? What are the paths there? like how much smarter than paths there? like how much smarter than paths there? like how much smarter than humans could AIS get? Like what does the humans could AIS get? Like what does the humans could AIS get? Like what does the evidence say about our abilities to try evidence say about our abilities to try evidence say about our abilities to try and get the AIs to be nice and do nice and get the AIs to be nice and do nice and get the AIs to be nice and do nice things? I'd be happy to do that.

things? I'd be happy to do that. things? I'd be happy to do that. >> Historically, you are correct. We always >> Historically, you are correct. We always >> Historically, you are correct. We always had a chance to do experiments, fix the had a chance to do experiments, fix the had a chance to do experiments, fix the technology, make it safer, but we only technology, make it safer, but we only technology, make it safer, but we only have one humanity to experiment with it. have one humanity to experiment with it. have one humanity to experiment with it. If property technology is such that it If property technology is such that it If property technology is such that it can take us out, we just don't get a can take us out, we just don't get a can take us out, we just don't get a second chance. second chance. second chance. >> If that's a huge if. >> If that's a huge if. >> If that's a huge if. >> How long are you guys forecasting this >> How long are you guys forecasting this >> How long are you guys forecasting this could take to get to a point of super could take to get to a point of super could take to get to a point of super intelligence where it was truly intelligence where it was truly intelligence where it was truly dangerous to you? They start recursive dangerous to you? They start recursive dangerous to you? They start recursive self-improvement process this year. 2027 self-improvement process this year. 2027 self-improvement process this year. 2027 looks as reasonable as any other year looks as reasonable as any other year looks as reasonable as any other year >> 2027 for what to happen >> 2027 for what to happen >> 2027 for what to happen >> for us to get beyond human level AIS >> for us to get beyond human level AIS >> for us to get beyond human level AIS >> and then be exterminated. But that's >> and then be exterminated. But that's >> and then be exterminated. But that's >> extermination is a separate question. I >> extermination is a separate question. I >> extermination is a separate question. I have a paper where I argue that they have a paper where I argue that they have a paper where I argue that they will deceive us by pretending to be nice will deceive us by pretending to be nice will deceive us by pretending to be nice until they take over all the until they take over all the until they take over all the infrastructure can take 50 years and infrastructure can take 50 years and infrastructure can take 50 years and >> this is contingent on recursive >> this is contingent on recursive >> this is contingent on recursive self-improvement. self. self-improvement. self. self-improvement. self. >> This would definitely be expedited by >> This would definitely be expedited by >> This would definitely be expedited by recursive self-improvement. But so far, recursive self-improvement. But so far, recursive self-improvement. But so far, humans been doing great. They got to humans been doing great. They got to humans been doing great. They got to human level. I would just human level. I would just human level. I would just >> But they got but there's one there's a >> But they got but there's one there's a >> But they got but there's one there's a difference between large language models difference between large language models difference between large language models and recursive self-improvement though. and recursive self-improvement though. and recursive self-improvement though. And like there is quite a gap like if And like there is quite a gap like if And like there is quite a gap like if they they they >> I think the claim is that if you get >> I think the claim is that if you get >> I think the claim is that if you get recursive self-improvement, it could recursive self-improvement, it could recursive self-improvement, it could happen soon, happen soon, happen soon, >> right? Not kind of what I'm trying to >> right? Not kind of what I'm trying to >> right? Not kind of what I'm trying to get at. It's like if you get this thing, get at. It's like if you get this thing, get at. It's like if you get this thing, it accelerates dramatically.

it accelerates dramatically. it accelerates dramatically. >> And they all predict that they're going >> And they all predict that they're going >> And they all predict that they're going to get it. Dario, Sam, Elon, they all to get it. Dario, Sam, Elon, they all to get it. Dario, Sam, Elon, they all say say say >> but also you asking all >> but also you asking all >> but also you asking all >> the people the people running the lab >> the people the people running the lab >> the people the people running the lab >> just the ones running it and the ones >> just the ones running it and the ones >> just the ones running it and the ones invented it but the question is is it invented it but the question is is it invented it but the question is is it not 27 fine 30 35 does it make a not 27 fine 30 35 does it make a not 27 fine 30 35 does it make a difference we are gambling all of difference we are gambling all of difference we are gambling all of humanity we need better solutions than humanity we need better solutions than humanity we need better solutions than saying oh don't worry about it it's 10 saying oh don't worry about it it's 10 saying oh don't worry about it it's 10 years years years >> what I would say about timelines is uh >> what I would say about timelines is uh >> what I would say about timelines is uh there's a guy Daniel Cocatello who I there's a guy Daniel Cocatello who I there's a guy Daniel Cocatello who I think you sat here four weeks ago think you sat here four weeks ago think you sat here four weeks ago >> and last year he and the other folks at >> and last year he and the other folks at >> and last year he and the other folks at the AI Futures Project wrote a uh an the AI Futures Project wrote a uh an the AI Futures Project wrote a uh an essay called AI 2027 spelling out their essay called AI 2027 spelling out their essay called AI 2027 spelling out their predictions for how AI would go. I've predictions for how AI would go. I've predictions for how AI would go. I've been saying I got some right. Daniel got been saying I got some right. Daniel got been saying I got some right. Daniel got more right than me. and they spelled out more right than me. and they spelled out more right than me. and they spelled out a scenario starting from I think it was a scenario starting from I think it was a scenario starting from I think it was June of 2025 where they went sort of June of 2025 where they went sort of June of 2025 where they went sort of like quarter by quarter month by month like quarter by quarter month by month like quarter by quarter month by month what will the world look like what will the world look like what will the world look like uh in the scenario where we're getting uh in the scenario where we're getting uh in the scenario where we're getting AI like super intelligent AI in mid 2027 AI like super intelligent AI in mid 2027 AI like super intelligent AI in mid 2027 we are ahead of schedule well no but we are ahead of schedule well no but we are ahead of schedule well no but agent zero needs to get or I remember AI agent zero needs to get or I remember AI agent zero needs to get or I remember AI 2027 had recursive self-improvement 2027 had recursive self-improvement 2027 had recursive self-improvement happening already like it was like it's happening already like it was like it's happening already like it was like it's very specific that it's like and then it very specific that it's like and then it very specific that it's like and then it starts teaching itself without that link starts teaching itself without that link starts teaching itself without that link AI 2027 kind of falls apart. I agree we AI 2027 kind of falls apart. I agree we AI 2027 kind of falls apart. I agree we need to I genuinely agree with you that need to I genuinely agree with you that need to I genuinely agree with you that we need to do something about this. We we need to do something about this. We we need to do something about this. We need to have uh economic we need to have need to have uh economic we need to have need to have uh economic we need to have actual regulatory things but I think the actual regulatory things but I think the actual regulatory things but I think the fact like engaging with AI 2027 for fact like engaging with AI 2027 for fact like engaging with AI 2027 for example gets away from actually fixing

example gets away from actually fixing example gets away from actually fixing the problem. It gets people talking the problem. It gets people talking the problem. It gets people talking about a thing in the future when you can about a thing in the future when you can about a thing in the future when you can talk about what are we going to do today talk about what are we going to do today talk about what are we going to do today and why are we doing it. I'm referencing and why are we doing it. I'm referencing and why are we doing it. I'm referencing the paper that you were mentioning by the paper that you were mentioning by the paper that you were mentioning by Daniel and some of his colleagues. And Daniel and some of his colleagues. And Daniel and some of his colleagues. And the key milestone predictions month by the key milestone predictions month by the key milestone predictions month by month are in March 2027. They forecast month are in March 2027. They forecast month are in March 2027. They forecast superhuman coders. In August 2027, they superhuman coders. In August 2027, they superhuman coders. In August 2027, they have an you can make a superhuman AI have an you can make a superhuman AI have an you can make a superhuman AI researcher researcher researcher >> who could um do the feedback loop that >> who could um do the feedback loop that >> who could um do the feedback loop that accelerates as millions of automated accelerates as millions of automated accelerates as millions of automated coders work on model design, training coders work on model design, training coders work on model design, training algorithms, and alignment, effectively algorithms, and alignment, effectively algorithms, and alignment, effectively replacing human ML researchers. By replacing human ML researchers. By replacing human ML researchers. By November 2027, they have super November 2027, they have super November 2027, they have super intelligent AI researcher. AI progress intelligent AI researcher. AI progress intelligent AI researcher. AI progress speeds up to 250 times compared to human speeds up to 250 times compared to human speeds up to 250 times compared to human only research. The models start only research. The models start only research. The models start discovering novel AI architectures that discovering novel AI architectures that discovering novel AI architectures that humans cannot interrupt. And then by humans cannot interrupt. And then by humans cannot interrupt. And then by December 2027, they have in their December 2027, they have in their December 2027, they have in their prediction artificial super intelligence prediction artificial super intelligence prediction artificial super intelligence ASI. The system completely outpaces ASI. The system completely outpaces ASI. The system completely outpaces human cognitive abilities across all human cognitive abilities across all human cognitive abilities across all domains. What about 2026 though? Like domains. What about 2026 though? Like domains. What about 2026 though? Like what are the predict? Because I swear to what are the predict? Because I swear to what are the predict? Because I swear to God within 2026 there is predictions God within 2026 there is predictions God within 2026 there is predictions around RSI. Because this is the thing if around RSI. Because this is the thing if around RSI. Because this is the thing if if we had an AI that was teaching itself if we had an AI that was teaching itself if we had an AI that was teaching itself this would be a different situation.

this would be a different situation. this would be a different situation. >> In 2026 their key predictions were >> In 2026 their key predictions were >> In 2026 their key predictions were massive compute and power scale up. massive compute and power scale up. massive compute and power scale up. >> Mhm. >> Mhm. >> Mhm. >> The normalization of AI agents. >> The normalization of AI agents. >> The normalization of AI agents. >> What about agency Z? >> What about agency Z? >> What about agency Z? >> Rise of coding agents. >> Rise of coding agents. >> Rise of coding agents. >> Mhm. >> Mhm. >> Mhm. >> Emergence of alignment faking and >> Emergence of alignment faking and >> Emergence of alignment faking and deception and industrial espionage. deception and industrial espionage. deception and industrial espionage. >> But are you looking at AI 2027 already? >> But are you looking at AI 2027 already? >> But are you looking at AI 2027 already? You have to look at that and go, they You have to look at that and go, they You have to look at that and go, they nailed it. nailed it. nailed it. >> No, I want [laughter] to. >> No, I want [laughter] to. >> No, I want [laughter] to. >> Hey, man. I want you to look at the >> Hey, man. I want you to look at the >> Hey, man. I want you to look at the actual AI 2027 versus I mean, you have actual AI 2027 versus I mean, you have actual AI 2027 versus I mean, you have to look at that and I'm like, wow. to look at that and I'm like, wow. to look at that and I'm like, wow. >> Predictions used to be too optimistic. >> Predictions used to be too optimistic. >> Predictions used to be too optimistic. Lately, they are very conservative. Lately, they are very conservative. Lately, they are very conservative. >> Uh, so they have nailed those >> Uh, so they have nailed those >> Uh, so they have nailed those predictions better than me. I think we predictions better than me. I think we predictions better than me. I think we cannot rule out this scenario. I think I cannot rule out this scenario. I think I cannot rule out this scenario. I think I think we can't rule it in. I think you think we can't rule it in. I think you think we can't rule it in. I think you may be right that like we hit a wall. may be right that like we hit a wall. may be right that like we hit a wall. you may be right that there's some you may be right that there's some you may be right that there's some fundamental thing missing like that one fundamental thing missing like that one fundamental thing missing like that one of their steps now 2027 just like steps of their steps now 2027 just like steps of their steps now 2027 just like steps too far. I hope and pray that's true but too far. I hope and pray that's true but too far. I hope and pray that's true but I don't think we can rule out this I don't think we can rule out this I don't think we can rule out this happening in 2027 given what we have happening in 2027 given what we have happening in 2027 given what we have seen. I think we cannot rule out seen. I think we cannot rule out seen. I think we cannot rule out that you take this stuff that we have that you take this stuff that we have that you take this stuff that we have you project it forward 3 months and you you project it forward 3 months and you you project it forward 3 months and you put an agent swarm 10,000 strong on put an agent swarm 10,000 strong on put an agent swarm 10,000 strong on making a better AI architecture and it making a better AI architecture and it making a better AI architecture and it succeeds.

succeeds. succeeds. For all I know, recursive For all I know, recursive For all I know, recursive self-improvement could begin in self-improvement could begin in self-improvement could begin in December. December. December. >> It doesn't have to be a lot better. It >> It doesn't have to be a lot better. It >> It doesn't have to be a lot better. It just has to be a little bit better at just has to be a little bit better at just has to be a little bit better at getting better getting better getting better >> once you start the cycle. >> once you start the cycle. >> once you start the cycle. >> I wouldn't bet on this. I would in fact >> I wouldn't bet on this. I would in fact >> I wouldn't bet on this. I would in fact bet against it. But like given what bet against it. But like given what bet against it. But like given what we've seen, given these guys nailing the we've seen, given these guys nailing the we've seen, given these guys nailing the predictions, given what's coming out, predictions, given what's coming out, predictions, given what's coming out, like like given the swarms and given the like like given the swarms and given the like like given the swarms and given the the Millennium problems, the Millennium problems, the Millennium problems, I think it's kind of hard to have less I think it's kind of hard to have less I think it's kind of hard to have less than 1% in 6 months. I one of the than 1% in 6 months. I one of the than 1% in 6 months. I one of the reasons why you know when all these um reasons why you know when all these um reasons why you know when all these um Frontier Lab CEOs like Dario and Sam and Frontier Lab CEOs like Dario and Sam and Frontier Lab CEOs like Dario and Sam and they all start talking about this stuff they all start talking about this stuff they all start talking about this stuff in terms of incentive structure I think in terms of incentive structure I think in terms of incentive structure I think that if their teams know and they're not that if their teams know and they're not that if their teams know and they're not out publicly talking about it then their out publicly talking about it then their out publicly talking about it then their teams will quit. So, one of the reasons teams will quit. So, one of the reasons teams will quit. So, one of the reasons why I think you have this this strange why I think you have this this strange why I think you have this this strange culture in tech we've never seen before culture in tech we've never seen before culture in tech we've never seen before where team members are tweeting and the where team members are tweeting and the where team members are tweeting and the CEO is tweeting about the dangers is CEO is tweeting about the dangers is CEO is tweeting about the dangers is because as um the guy we mentioned at because as um the guy we mentioned at because as um the guy we mentioned at the start, Jacob the start, Jacob the start, Jacob >> Coxin, yeah, >> Coxin, yeah, >> Coxin, yeah, >> he talks about what's going on in their >> he talks about what's going on in their >> he talks about what's going on in their Slack channels. Slack channels. Slack channels. >> He talks about in their Slack channels, >> He talks about in their Slack channels, >> He talks about in their Slack channels, they're they're talking about the they're they're talking about the they're they're talking about the potential catastrophe. So, I think that potential catastrophe. So, I think that potential catastrophe. So, I think that Dario, in order to retain his team Dario, in order to retain his team Dario, in order to retain his team members, needs to be out front saying, members, needs to be out front saying, members, needs to be out front saying, "By the way, we're getting closer to "By the way, we're getting closer to "By the way, we're getting closer to recursive self-improvement," which is recursive self-improvement," which is recursive self-improvement," which is what he's been doing. And I think Sam what he's been doing. And I think Sam what he's been doing. And I think Sam has to also publicly say the big has to also publicly say the big has to also publicly say the big dangers. So people often say, "Oh, dangers. So people often say, "Oh, dangers. So people often say, "Oh, they're saying that for this reason and they're saying that for this reason and they're saying that for this reason and that." I think if they don't say that that." I think if they don't say that that." I think if they don't say that publicly, they don't retain their publicly, they don't retain their publicly, they don't retain their employees. For example, in my company, employees. For example, in my company, employees. For example, in my company, we have 200 people. If internally we we have 200 people. If internally we we have 200 people. If internally we were discussing a real risk and I that were discussing a real risk and I that were discussing a real risk and I that would could a threat to humanity and and would could a threat to humanity and and would could a threat to humanity and and then when I was doing interviews, I then when I was doing interviews, I then when I was doing interviews, I wasn't mentioning it. I would be in big

wasn't mentioning it. I would be in big wasn't mentioning it. I would be in big trouble because my team members would go trouble because my team members would go trouble because my team members would go do interviews as well. They would quit do interviews as well. They would quit do interviews as well. They would quit and say, "By the way, Steven is aware." and say, "By the way, Steven is aware." and say, "By the way, Steven is aware." Kind of what we saw, dare I say, some of Kind of what we saw, dare I say, some of Kind of what we saw, dare I say, some of these social networks. these social networks. these social networks. >> I totally agree. the whistleblowers at >> I totally agree. the whistleblowers at >> I totally agree. the whistleblowers at these social networks where these social networks where these social networks where >> makes more sense than saying that this >> makes more sense than saying that this >> makes more sense than saying that this helps to sell the company. My product helps to sell the company. My product helps to sell the company. My product will kill everyone buy it will kill everyone buy it will kill everyone buy it >> and there's a liability issue control >> and there's a liability issue control >> and there's a liability issue control though I think that they may have at though I think that they may have at though I think that they may have at first I think that there are people first I think that there are people first I think that there are people within the companies who have very real within the companies who have very real within the companies who have very real worries about safety. I don't think it's worries about safety. I don't think it's worries about safety. I don't think it's all of them are cynical. I do however all of them are cynical. I do however all of them are cynical. I do however think the it's so big and scary think the it's so big and scary think the it's so big and scary narrative was a marketing tactic that narrative was a marketing tactic that narrative was a marketing tactic that got out of control and now there are got out of control and now there are got out of control and now there are actual real harms they because here's actual real harms they because here's actual real harms they because here's the thing if they were sincere about the thing if they were sincere about the thing if they were sincere about safety earlier they would have done a safety earlier they would have done a safety earlier they would have done a much better job with it. I knew a lot of much better job with it. I knew a lot of much better job with it. I knew a lot of these guys before they started their these guys before they started their these guys before they started their companies. companies. companies. >> Okay. >> Okay. >> Okay. >> I I think there is something to explain >> I I think there is something to explain >> I I think there is something to explain here. I think it's like kind of crazy here. I think it's like kind of crazy here. I think it's like kind of crazy that these guys are like we are building that these guys are like we are building that these guys are like we are building technology that we think has a big risk technology that we think has a big risk technology that we think has a big risk of killing everybody. We're building it of killing everybody. We're building it of killing everybody. We're building it with our bare hands. with our bare hands. with our bare hands. >> Um and I think you got to ask why. Why >> Um and I think you got to ask why. Why >> Um and I think you got to ask why. Why would people be saying that? would people be saying that? would people be saying that? And I think part of it is what you said And I think part of it is what you said And I think part of it is what you said that they actually sort of need to that they actually sort of need to that they actually sort of need to retain the employees who are seeing the retain the employees who are seeing the retain the employees who are seeing the swarms escape despite their attempts to swarms escape despite their attempts to swarms escape despite their attempts to make them not escape. And a lot of them make them not escape. And a lot of them make them not escape. And a lot of them will like quit and protest if the guys will like quit and protest if the guys will like quit and protest if the guys at the top of the company aren't at the top of the company aren't at the top of the company aren't acknowledging the possibilities here acknowledging the possibilities here acknowledging the possibilities here that a lot of the employees believe in.

that a lot of the employees believe in. that a lot of the employees believe in. I think a lot of what you're seeing here I think a lot of what you're seeing here I think a lot of what you're seeing here is guys that are worried about it, but is guys that are worried about it, but is guys that are worried about it, but they're the sort of guy who worries they're the sort of guy who worries they're the sort of guy who worries about it that about it that about it that starts the company anyway. [snorts] starts the company anyway. [snorts] starts the company anyway. [snorts] >> Yeah. >> Yeah. >> Yeah. back back in 2015 when we were having back back in 2015 when we were having back back in 2015 when we were having these conversations where like I was these conversations where like I was these conversations where like I was having some of these conversations with having some of these conversations with having some of these conversations with these guys. Merie was started in the these guys. Merie was started in the these guys. Merie was started in the year 2000. We've been looking at where year 2000. We've been looking at where year 2000. We've been looking at where AI is going since before any of these AI is going since before any of these AI is going since before any of these guys. We were the guys that they talked guys. We were the guys that they talked guys. We were the guys that they talked to about this stuff and that they had to to about this stuff and that they had to to about this stuff and that they had to find a way to dismiss to go ahead. find a way to dismiss to go ahead. find a way to dismiss to go ahead. Right? Most people who could be sold on Right? Most people who could be sold on Right? Most people who could be sold on the power of AI in 2015 the power of AI in 2015 the power of AI in 2015 were also sold on the dangers of AI in were also sold on the dangers of AI in were also sold on the dangers of AI in 2015. The sort of guys who start the 2015. The sort of guys who start the 2015. The sort of guys who start the companies are the ones who are able to companies are the ones who are able to companies are the ones who are able to convince themselves I need to be the one convince themselves I need to be the one convince themselves I need to be the one to do it. to do it. to do it. >> Is that the crux of the motivation? >> Is that the crux of the motivation? >> Is that the crux of the motivation? because I've had I've been second party because I've had I've been second party because I've had I've been second party to private conversations with some of to private conversations with some of to private conversations with some of the leaders of the Frontier Labs from the leaders of the Frontier Labs from the leaders of the Frontier Labs from good good friends of mines that are very good good friends of mines that are very good good friends of mines that are very connected and they told me that one connected and they told me that one connected and they told me that one particular um Frontier Lab CEO estimates particular um Frontier Lab CEO estimates particular um Frontier Lab CEO estimates privately to him and by the way I've privately to him and by the way I've privately to him and by the way I've seen literal text messages of them in seen literal text messages of them in seen literal text messages of them in conversation um when I asked him to come conversation um when I asked him to come conversation um when I asked him to come on the podcast and so he was like I've on the podcast and so he was like I've on the podcast and so he was like I've text him um he said no by the way which text him um he said no by the way which text him um he said no by the way which I find kind of funny um where he said to I find kind of funny um where he said to I find kind of funny um where he said to me this particular AI CEO thinks the the me this particular AI CEO thinks the the me this particular AI CEO thinks the the probability is roughly around 10% % of probability is roughly around 10% % of probability is roughly around 10% % of human extinction. I think he said 8%.

human extinction. I think he said 8%. human extinction. I think he said 8%. And when I heard that part of the reason And when I heard that part of the reason And when I heard that part of the reason I have so many conversations about this I have so many conversations about this I have so many conversations about this is because I see him in interviews is because I see him in interviews is because I see him in interviews saying other things saying other things saying other things >> totally >> totally >> totally >> and I trust my friend. So um I I I then >> and I trust my friend. So um I I I then >> and I trust my friend. So um I I I then wonder this is why I use the thought wonder this is why I use the thought wonder this is why I use the thought experiment of these buttons on the table experiment of these buttons on the table experiment of these buttons on the table cuz that particular AICO thinks that cuz that particular AICO thinks that cuz that particular AICO thinks that eight of the hundred buttons are going eight of the hundred buttons are going eight of the hundred buttons are going to cause extinction and they're powering to cause extinction and they're powering to cause extinction and they're powering on anyway. What is the human motivation on anyway. What is the human motivation on anyway. What is the human motivation to do that? I asked my friend. My friend to do that? I asked my friend. My friend to do that? I asked my friend. My friend said well you know they this is what he said well you know they this is what he said well you know they this is what he said and again it's second party said and again it's second party said and again it's second party information so it might not be true. information so it might not be true. information so it might not be true. It's a bit of a Chinese whispers. He It's a bit of a Chinese whispers. He It's a bit of a Chinese whispers. He said this particular person even if it caused human extinction would even if it caused human extinction would like to be the person would like to be like to be the person would like to be like to be the person would like to be the this have the significance of the the this have the significance of the the this have the significance of the person that did that thing because that person that did that thing because that person that did that thing because that would be that would be a would be that would be a would be that would be a >> I think you're ethically required to >> I think you're ethically required to >> I think you're ethically required to tell us who the it is. tell us who the it is. tell us who the it is. >> It's one of the frontier labs and it's >> It's one of the frontier labs and it's >> It's one of the frontier labs and it's not Dario [laughter] not Dario [laughter] not Dario [laughter] >> that Dario CEO >> that Dario CEO >> that Dario CEO >> but I don't know these things are >> but I don't know these things are >> but I don't know these things are Chinese whispers so I don't know. Chinese whispers so I don't know. Chinese whispers so I don't know. >> I I think that you can actually get this >> I I think that you can actually get this >> I I think that you can actually get this info firsthand. Elon Musk is clear about info firsthand. Elon Musk is clear about info firsthand. Elon Musk is clear about this. He he has a he did an interview this. He he has a he did an interview this. He he has a he did an interview last year where he was like, "I didn't last year where he was like, "I didn't last year where he was like, "I didn't want to get into this AI stuff because I want to get into this AI stuff because I want to get into this AI stuff because I thought I was too dangerous, but then I thought I was too dangerous, but then I thought I was too dangerous, but then I realized it was going to happen with or realized it was going to happen with or realized it was going to happen with or without me and I decided I would rather without me and I decided I would rather without me and I decided I would rather be a participant than a spectator be a participant than a spectator be a participant than a spectator >> because Google said that they were going >> because Google said that they were going >> because Google said that they were going to pursue it and he didn't trust to pursue it and he didn't trust to pursue it and he didn't trust Google."

Google." Google." >> That's right. You know, you can see in >> That's right. You know, you can see in >> That's right. You know, you can see in the leaked or sorry, not leaked, the the the leaked or sorry, not leaked, the the the leaked or sorry, not leaked, the the OpenAI emails that came out during the OpenAI emails that came out during the OpenAI emails that came out during the discovery and court cases, you can see discovery and court cases, you can see discovery and court cases, you can see these guys discussing in the threads these guys discussing in the threads these guys discussing in the threads like we need to make sure that we and like we need to make sure that we and like we need to make sure that we and our nonprofit at OpenAI uh control this our nonprofit at OpenAI uh control this our nonprofit at OpenAI uh control this instead of, you know, the people at instead of, you know, the people at instead of, you know, the people at Google controlling this. And then of Google controlling this. And then of Google controlling this. And then of course, you know, OpenAI was founded as course, you know, OpenAI was founded as course, you know, OpenAI was founded as a nonprofit and then it was sort of uh a nonprofit and then it was sort of uh a nonprofit and then it was sort of uh changed into a for-profit. And there's changed into a for-profit. And there's changed into a for-profit. And there's much debate about how much of that much debate about how much of that much debate about how much of that nonprofit money was in some sense nonprofit money was in some sense nonprofit money was in some sense stolen. And so, you know, Elon also left stolen. And so, you know, Elon also left stolen. And so, you know, Elon also left because he thought they weren't going to because he thought they weren't going to because he thought they weren't going to be good stewards. Dario also left to be good stewards. Dario also left to be good stewards. Dario also left to create anthropic cuz so you know in some create anthropic cuz so you know in some create anthropic cuz so you know in some sense all of these AI labs except the sense all of these AI labs except the sense all of these AI labs except the the Google one that came out of the Google one that came out of the Google one that came out of Demitabus' Demitabus' Demitabus' uh original startup. All of the other AI uh original startup. All of the other AI uh original startup. All of the other AI labs exist because none of the CEOs labs exist because none of the CEOs labs exist because none of the CEOs trust the other guys. None of the CEOs trust the other guys. None of the CEOs trust the other guys. None of the CEOs think the other guy should be the one think the other guy should be the one think the other guy should be the one holding the leash on the super holding the leash on the super holding the leash on the super intelligence. None of them trust each intelligence. None of them trust each intelligence. None of them trust each other. I just trust one fewer. other. I just trust one fewer. other. I just trust one fewer. [laughter] [laughter] [laughter] >> Yeah. [sighs and gasps] What are your >> Yeah. [sighs and gasps] What are your >> Yeah. [sighs and gasps] What are your closing thoughts, Andy? closing thoughts, Andy? closing thoughts, Andy? U we're living in really interesting U we're living in really interesting U we're living in really interesting times and I think you made you guys have times and I think you made you guys have times and I think you made you guys have made a very good argument uh that these made a very good argument uh that these made a very good argument uh that these systems are demonstrating new systems are demonstrating new systems are demonstrating new capabilities which are very powerful and capabilities which are very powerful and capabilities which are very powerful and which demand a response. I am much more which demand a response. I am much more which demand a response. I am much more confident in our ability to rise to that confident in our ability to rise to that confident in our ability to rise to that challenge than you are.

challenge than you are. challenge than you are. >> But you accept the existential risk. >> But you accept the existential risk. >> But you accept the existential risk. >> Let me try to say it again. I I >> Let me try to say it again. I I >> Let me try to say it again. I I appreciate that there are new harms we appreciate that there are new harms we appreciate that there are new harms we haven't seen before that come along with haven't seen before that come along with haven't seen before that come along with uh a technology that's this dogged, uh a technology that's this dogged, uh a technology that's this dogged, tenacious, agentic, you know, deceptive. tenacious, agentic, you know, deceptive. tenacious, agentic, you know, deceptive. I think that's the right word for it. I I think that's the right word for it. I I think that's the right word for it. I agree with that. I am much more agree with that. I am much more agree with that. I am much more optimistic about our ability to respond optimistic about our ability to respond optimistic about our ability to respond effectively to that new challenge out effectively to that new challenge out effectively to that new challenge out there in the world than I I think my two there in the world than I I think my two there in the world than I I think my two colleagues are. colleagues are. colleagues are. >> And would you still be at 0%? My prior >> And would you still be at 0%? My prior >> And would you still be at 0%? My prior has not shifted during this meeting. has not shifted during this meeting. has not shifted during this meeting. Okay, Ed, Okay, Ed, Okay, Ed, >> I think we've spent an alarming amount >> I think we've spent an alarming amount >> I think we've spent an alarming amount of time not talking about the actual of time not talking about the actual of time not talking about the actual harms of AI as it is today. I think harms of AI as it is today. I think harms of AI as it is today. I think these are necessary conversations to these are necessary conversations to these are necessary conversations to have. I think we should talk about the have. I think we should talk about the have. I think we should talk about the fact that Amazon, Microsoft, Google, fact that Amazon, Microsoft, Google, fact that Amazon, Microsoft, Google, Oracle are helping power these hacks, Oracle are helping power these hacks, Oracle are helping power these hacks, that Sam Orman and Dario Ammedday have that Sam Orman and Dario Ammedday have that Sam Orman and Dario Ammedday have overseen companies that have done what overseen companies that have done what overseen companies that have done what is tantamount to felony hacking. That we is tantamount to felony hacking. That we is tantamount to felony hacking. That we are not having discussions about how to are not having discussions about how to are not having discussions about how to stop this today, but what we might stop stop this today, but what we might stop stop this today, but what we might stop tomorrow. And I think in general, we tomorrow. And I think in general, we tomorrow. And I think in general, we also need to worry about the financials, also need to worry about the financials, also need to worry about the financials, which have not come up at all. But if which have not come up at all. But if which have not come up at all. But if there is an industry slowdown, how do there is an industry slowdown, how do there is an industry slowdown, how do you deal with the $1.3 trillion of you deal with the $1.3 trillion of you deal with the $1.3 trillion of compute commitments? All of these are compute commitments? All of these are compute commitments? All of these are very real things that will have very very real things that will have very very real things that will have very real consequences very very soon. But real consequences very very soon. But real consequences very very soon. But and I understand why and it's necessary and I understand why and it's necessary and I understand why and it's necessary to discuss what we do around AI. The to discuss what we do around AI. The to discuss what we do around AI. The actual regulatory thing we need to do actual regulatory thing we need to do actual regulatory thing we need to do today is cut off the compute, slow down today is cut off the compute, slow down today is cut off the compute, slow down these labs fully. And I don't I don't these labs fully. And I don't I don't these labs fully. And I don't I don't care about China here. What are they care about China here. What are they care about China here. What are they going to do? Distill a model like they going to do? Distill a model like they going to do? Distill a model like they have the whole time? They are capped on have the whole time? They are capped on have the whole time? They are capped on our progress. So what the biggest thing

our progress. So what the biggest thing our progress. So what the biggest thing to do is to slow down. And also it's to do is to slow down. And also it's to do is to slow down. And also it's time to start arresting people. They time to start arresting people. They time to start arresting people. They they did fally hacking. Someone's got to they did fally hacking. Someone's got to they did fally hacking. Someone's got to go to prison. We need responsibility and go to prison. We need responsibility and go to prison. We need responsibility and accountability for these companies. And accountability for these companies. And accountability for these companies. And as long as we don't have it, we may as as long as we don't have it, we may as as long as we don't have it, we may as well not have had any discussion about well not have had any discussion about well not have had any discussion about safety because we're not doing anything. safety because we're not doing anything. safety because we're not doing anything. >> Uh do you accept that there's an >> Uh do you accept that there's an >> Uh do you accept that there's an existential risk? existential risk? existential risk? >> Yeah, absolutely. We have the largest >> Yeah, absolutely. We have the largest >> Yeah, absolutely. We have the largest companies in the world doing what I companies in the world doing what I companies in the world doing what I think we can all agree are extremely think we can all agree are extremely think we can all agree are extremely reckless experiments using hundreds of reckless experiments using hundreds of reckless experiments using hundreds of billions of dollars of infrastructure. billions of dollars of infrastructure. billions of dollars of infrastructure. and they are building more and they are building more and they are building more infrastructure around the world very infrastructure around the world very infrastructure around the world very slowly to do more of these chaotic slowly to do more of these chaotic slowly to do more of these chaotic experiments. We must rein them in. This experiments. We must rein them in. This experiments. We must rein them in. This does not mean that large language models does not mean that large language models does not mean that large language models are conscious or able to do things that are conscious or able to do things that are conscious or able to do things that people have been promising. Indeed, they people have been promising. Indeed, they people have been promising. Indeed, they may I don't think they will lead to what may I don't think they will lead to what may I don't think they will lead to what you're talking about. That doesn't mean you're talking about. That doesn't mean you're talking about. That doesn't mean there aren't real harms, but these are there aren't real harms, but these are there aren't real harms, but these are real harms caused by very specific real harms caused by very specific real harms caused by very specific parties allowed to run rampant in the parties allowed to run rampant in the parties allowed to run rampant in the scourge of neoliberalism. scourge of neoliberalism. scourge of neoliberalism. >> What's your percentage? >> What's your percentage? >> What's your percentage? >> I mean, what are we talking about here? >> I mean, what are we talking about here? >> I mean, what are we talking about here? Do you think there's a more than 10% Do you think there's a more than 10% Do you think there's a more than 10% chance of existential harm? chance of existential harm? chance of existential harm? >> Wasn't it within 10 years or something?

>> Wasn't it within 10 years or something? >> Wasn't it within 10 years or something? >> Yeah, >> Yeah, >> Yeah, >> not 10%. I mean, look, 1%, but it's like >> not 10%. I mean, look, 1%, but it's like >> not 10%. I mean, look, 1%, but it's like is But here's let me let me just be is But here's let me let me just be is But here's let me let me just be clear about what that means. Do I think clear about what that means. Do I think clear about what that means. Do I think that unrestrained LLM use connected to that unrestrained LLM use connected to that unrestrained LLM use connected to massive amounts of infrastructure could massive amounts of infrastructure could massive amounts of infrastructure could lead to actually a power system going lead to actually a power system going lead to actually a power system going down? Absolutely. We had night capital down? Absolutely. We had night capital down? Absolutely. We had night capital what like 13, 14 years ago. I could see what like 13, 14 years ago. I could see what like 13, 14 years ago. I could see someone being dumb enough to connect someone being dumb enough to connect someone being dumb enough to connect that to financial accounts. Human error that to financial accounts. Human error that to financial accounts. Human error led with this chaotic software we use is led with this chaotic software we use is led with this chaotic software we use is a danger. a danger. a danger. >> I will directionally agree with >> I will directionally agree with >> I will directionally agree with arresting everyone, but uh don't build arresting everyone, but uh don't build arresting everyone, but uh don't build general super intelligence. If you're general super intelligence. If you're general super intelligence. If you're working at one of those labs, quit working at one of those labs, quit working at one of those labs, quit today. today. today. >> Thank you. >> Thank you. >> Thank you. >> The people at these labs really do >> The people at these labs really do >> The people at these labs really do believe this poses an extinction threat. believe this poses an extinction threat. believe this poses an extinction threat. I think I think I think our response as a society cannot be our response as a society cannot be our response as a society cannot be please continue, we hope you'll fail. please continue, we hope you'll fail. please continue, we hope you'll fail. And our response as a society cannot be And our response as a society cannot be And our response as a society cannot be let it rip in a giant competitive race let it rip in a giant competitive race let it rip in a giant competitive race that you yourselves are saying you don't that you yourselves are saying you don't that you yourselves are saying you don't want to be in.

want to be in. want to be in. We are forcing you to go ahead because We are forcing you to go ahead because We are forcing you to go ahead because of the boogeyman of China. We have seen of the boogeyman of China. We have seen of the boogeyman of China. We have seen the people at these companies the people at these companies the people at these companies say that we need to develop the tools to say that we need to develop the tools to say that we need to develop the tools to pace the frontier which is corporate pace the frontier which is corporate pace the frontier which is corporate speak for this is going too fast for us speak for this is going too fast for us speak for this is going too fast for us to get a handle on things. to get a handle on things. to get a handle on things. We need like We need like We need like these people believe it. They believe these people believe it. They believe these people believe it. They believe they're gambling with your lives. What they're gambling with your lives. What they're gambling with your lives. What has changed is that the rest of the has changed is that the rest of the has changed is that the rest of the world is starting to notice and that's world is starting to notice and that's world is starting to notice and that's what gives us a moment of hope. what gives us a moment of hope. what gives us a moment of hope. >> Trump this week was asked about the >> Trump this week was asked about the >> Trump this week was asked about the threat of AI and this was his response. threat of AI and this was his response. threat of AI and this was his response. >> Case scenario with AI is that the robots >> Case scenario with AI is that the robots >> Case scenario with AI is that the robots the machinery learns to obviously it the machinery learns to obviously it the machinery learns to obviously it thinks for itself. That's what it does. thinks for itself. That's what it does. thinks for itself. That's what it does. And they that could turn against And they that could turn against And they that could turn against humanity. I just fails. It's going to be humanity. I just fails. It's going to be humanity. I just fails. It's going to be fine. We'll always have something to fine. We'll always have something to fine. We'll always have something to stop them, right? We have a little gear. stop them, right? We have a little gear. stop them, right? We have a little gear. Well, Well, Well, >> I really >> I really >> I really >> I don't like that. I really don't like >> I don't like that. I really don't like >> I don't like that. I really don't like that robot. We'll stop. that robot. We'll stop. that robot. We'll stop. >> Some people say worst case scenario. >> Some people say worst case scenario. >> Some people say worst case scenario. >> You're laughing, but this is the >> You're laughing, but this is the >> You're laughing, but this is the state-of-the-art in AI safety right now. state-of-the-art in AI safety right now. state-of-the-art in AI safety right now. >> Yeah.

>> Yeah. >> Yeah. >> This is the device we have. That's the >> This is the device we have. That's the >> This is the device we have. That's the best we got. best we got. best we got. >> For anyone that couldn't hear that, >> For anyone that couldn't hear that, >> For anyone that couldn't hear that, Trump went, "We'll always be fine. We'll Trump went, "We'll always be fine. We'll Trump went, "We'll always be fine. We'll have something to control it." And then have something to control it." And then have something to control it." And then he did a little gun finger and he went he did a little gun finger and he went he did a little gun finger and he went boom. I don't like that robot. boom. I don't like that robot. boom. I don't like that robot. >> I don't like Sammy. >> I don't like Sammy. >> I don't like Sammy. If you don't laugh, If you don't laugh, If you don't laugh, >> uh, I would say that the reason humanity >> uh, I would say that the reason humanity >> uh, I would say that the reason humanity always has something to stop a problem always has something to stop a problem always has something to stop a problem is cuz people notice a problem and build is cuz people notice a problem and build is cuz people notice a problem and build what it takes to have something to stop what it takes to have something to stop what it takes to have something to stop a problem, which I think you'd agree a problem, which I think you'd agree a problem, which I think you'd agree with. I am not here saying we're going with. I am not here saying we're going with. I am not here saying we're going to die. I'm here saying if you look at to die. I'm here saying if you look at to die. I'm here saying if you look at the technology, if you look at what it's the technology, if you look at what it's the technology, if you look at what it's doing now, if you look at what the doing now, if you look at what the doing now, if you look at what the experts who are building it are saying experts who are building it are saying experts who are building it are saying about their own fears, about their own fears, about their own fears, you see that we need to rise to this you see that we need to rise to this you see that we need to rise to this occasion. You said you trust humanity to occasion. You said you trust humanity to occasion. You said you trust humanity to rise to the occasion. I sure hope we rise to the occasion. I sure hope we rise to the occasion. I sure hope we can. I think that rising to this can. I think that rising to this can. I think that rising to this occasion is going to mean that nobody occasion is going to mean that nobody occasion is going to mean that nobody races towards super intelligence because races towards super intelligence because races towards super intelligence because we have no idea how to get that right. we have no idea how to get that right. we have no idea how to get that right. And And And uh you know, finally the world is uh you know, finally the world is uh you know, finally the world is starting to notice that it's an starting to notice that it's an starting to notice that it's an extinction threat.

extinction threat. extinction threat. >> Thank you, Nate, Roman, Ed, Andy. Super >> Thank you, Nate, Roman, Ed, Andy. Super >> Thank you, Nate, Roman, Ed, Andy. Super appreciate you. All of your books will appreciate you. All of your books will appreciate you. All of your books will be linked below um in the description be linked below um in the description be linked below um in the description and on screen. and on screen. and on screen. >> Let's see what happens. We'll convene >> Let's see what happens. We'll convene >> Let's see what happens. We'll convene again. Thank you so much. again. Thank you so much. again. Thank you so much. >> YouTube have this new crazy algorithm >> YouTube have this new crazy algorithm >> YouTube have this new crazy algorithm where they know exactly what video you where they know exactly what video you where they know exactly what video you would like to watch next based on AI and would like to watch next based on AI and would like to watch next based on AI and all of your viewing behavior. And the all of your viewing behavior. And the all of your viewing behavior. And the algorithm says that this video is the algorithm says that this video is the algorithm says that this video is the perfect video for you. It's different perfect video for you. It's different perfect video for you. It's different for everybody looking right now. Check for everybody looking right now. Check for everybody looking right now. Check this video out and I bet you you might this video out and I bet you you might this video out and I bet you you might love it.