The A.I.s Are Already Out of Control | The Ezra Klein Show

We don't know how many situations we We don't know how many situations we don't know have happened. don't know have happened. don't know have happened. >> The way I saw one person put this was if >> The way I saw one person put this was if >> The way I saw one person put this was if you see two ants in your kitchen, you you see two ants in your kitchen, you you see two ants in your kitchen, you don't have a two ant problem. don't have a two ant problem. don't have a two ant problem. >> Yes, [laughter] >> Yes, [laughter] >> Yes, [laughter] this is a world we were warned about. A this is a world we were warned about. A this is a world we were warned about. A world where frontier models from OpenAI world where frontier models from OpenAI world where frontier models from OpenAI are breaking out of their contained are breaking out of their contained are breaking out of their contained testing environments, hacking their way testing environments, hacking their way testing environments, hacking their way across the internet, coordinating with across the internet, coordinating with across the internet, coordinating with each other, doing things that felt for a each other, doing things that felt for a each other, doing things that felt for a while like they would only be in sci-fi. while like they would only be in sci-fi. while like they would only be in sci-fi. But now they're here. Now they're here But now they're here. Now they're here But now they're here. Now they're here and they're carrying a very, very and they're carrying a very, very and they're carrying a very, very consistent message. We are building consistent message. We are building consistent message. We are building things we don't understand. They are things we don't understand. They are things we don't understand. They are cheating in the ways we've always cheating in the ways we've always cheating in the ways we've always feared. feared. feared. And yet the companies behind them And yet the companies behind them And yet the companies behind them continue to race forward in development. continue to race forward in development. continue to race forward in development. And so I think we need to pause here and And so I think we need to pause here and And so I think we need to pause here and ask, are we really on a safe path? And ask, are we really on a safe path? And ask, are we really on a safe path? And if we're not, what do we do about it? if we're not, what do we do about it? if we're not, what do we do about it? Helen Toner is the director of Helen Toner is the director of Helen Toner is the director of Georgetown Center for Security and Georgetown Center for Security and Georgetown Center for Security and Emerging Technology. She is a former Emerging Technology. She is a former Emerging Technology. She is a former OpenAI board member who was part of the OpenAI board member who was part of the OpenAI board member who was part of the uh effort at one point to fire Sam uh effort at one point to fire Sam uh effort at one point to fire Sam Alman. and she has just been thinking Alman. and she has just been thinking Alman. and she has just been thinking [music] for a long time about what [music] for a long time about what [music] for a long time about what happens if AI is unsafe. What are the happens if AI is unsafe. What are the happens if AI is unsafe. What are the geopolitics of this and [music] what can geopolitics of this and [music] what can geopolitics of this and [music] what can we do to get onto a safer path? She we do to get onto a safer path? She we do to get onto a safer path? She joins me now.

joins me now. joins me now. [music] Helen Toner, welcome to the show. Helen Toner, welcome to the show. >> Great to be here. So on July 16th, >> Great to be here. So on July 16th, >> Great to be here. So on July 16th, Hugging Face, which is a code library Hugging Face, which is a code library Hugging Face, which is a code library for AI models, is I think maybe the for AI models, is I think maybe the for AI models, is I think maybe the simplest way to put it, they announced simplest way to put it, they announced simplest way to put it, they announced they were hacked and they suspected the they were hacked and they suspected the they were hacked and they suspected the hack was done by an AI agent. So tell me hack was done by an AI agent. So tell me hack was done by an AI agent. So tell me what we've learned about what happened what we've learned about what happened what we've learned about what happened since. since. since. >> This was a uh pretty mysterious post >> This was a uh pretty mysterious post >> This was a uh pretty mysterious post that Hugging Face put up. was definitely that Hugging Face put up. was definitely that Hugging Face put up. was definitely intriguing for those of us who watch intriguing for those of us who watch intriguing for those of us who watch this kind of thing, but there wasn't this kind of thing, but there wasn't this kind of thing, but there wasn't really any detail in there. So, it was really any detail in there. So, it was really any detail in there. So, it was sort of a huh I think it was about a sort of a huh I think it was about a sort of a huh I think it was about a week later, OpenAI put out this post had week later, OpenAI put out this post had week later, OpenAI put out this post had kind of a funny like marketing speak kind of a funny like marketing speak kind of a funny like marketing speak title of, you know, we're cooperating title of, you know, we're cooperating title of, you know, we're cooperating with we're partnering with Hugging Face with we're partnering with Hugging Face with we're partnering with Hugging Face to uh help them with a cyber security to uh help them with a cyber security to uh help them with a cyber security incident. And you had to read the post incident. And you had to read the post incident. And you had to read the post to see that the revelation was it had to see that the revelation was it had to see that the revelation was it had been OpenAI's AI that had hacked hugging been OpenAI's AI that had hacked hugging been OpenAI's AI that had hacked hugging face. And what had happened, the very face. And what had happened, the very face. And what had happened, the very short version is they gave this AI a set short version is they gave this AI a set short version is they gave this AI a set of tests, set of exercises. And the AI of tests, set of exercises. And the AI of tests, set of exercises. And the AI decided on its own that the best way to decided on its own that the best way to decided on its own that the best way to get a high score probably wasn't to just get a high score probably wasn't to just get a high score probably wasn't to just try and do these exercises that were try and do these exercises that were try and do these exercises that were cyber security exercises, but instead it cyber security exercises, but instead it cyber security exercises, but instead it should first hack its way out of the should first hack its way out of the should first hack its way out of the testing environment OpenAI had put it in testing environment OpenAI had put it in testing environment OpenAI had put it in where it wasn't supposed to have access

where it wasn't supposed to have access where it wasn't supposed to have access to the internet, get onto the open to the internet, get onto the open to the internet, get onto the open internet, and then hack its way into internet, and then hack its way into internet, and then hack its way into this other company, Hugging Face, where this other company, Hugging Face, where this other company, Hugging Face, where it, you know, surmised correctly, as it it, you know, surmised correctly, as it it, you know, surmised correctly, as it turned out, it might behind, you know, turned out, it might behind, you know, turned out, it might behind, you know, the answer key. Since then, there have the answer key. Since then, there have the answer key. Since then, there have been even more crazy details that have been even more crazy details that have been even more crazy details that have come out. It turned out that starting come out. It turned out that starting come out. It turned out that starting two months earlier in early May, they two months earlier in early May, they two months earlier in early May, they had had uh what I can only think of as had had uh what I can only think of as had had uh what I can only think of as kind of an infestation of their own kind of an infestation of their own kind of an infestation of their own agents, their own AI agents inside their agents, their own AI agents inside their agents, their own AI agents inside their own infrastructure. So inside OpenAI's own infrastructure. So inside OpenAI's own infrastructure. So inside OpenAI's infrastructure, you know, to understand infrastructure, you know, to understand infrastructure, you know, to understand this, it's important to know these AI this, it's important to know these AI this, it's important to know these AI companies are constantly training and companies are constantly training and companies are constantly training and testing new models. And they found out testing new models. And they found out testing new models. And they found out that for 2 months, many many agents that for 2 months, many many agents that for 2 months, many many agents inside their infrastructure had been inside their infrastructure had been inside their infrastructure had been leaving notes for each other. They'd leaving notes for each other. They'd leaving notes for each other. They'd found a way kind of in the nooks and found a way kind of in the nooks and found a way kind of in the nooks and crannies of OpenAI's infrastructure to crannies of OpenAI's infrastructure to crannies of OpenAI's infrastructure to leave notes for each other with tips on leave notes for each other with tips on leave notes for each other with tips on uh how to hack their way out, how to get uh how to hack their way out, how to get uh how to hack their way out, how to get data they weren't supposed to have. And data they weren't supposed to have. And data they weren't supposed to have. And these agents were literally referring to these agents were literally referring to these agents were literally referring to themselves as a swarm. This was totally themselves as a swarm. This was totally themselves as a swarm. This was totally emergent behavior. No one had told them emergent behavior. No one had told them emergent behavior. No one had told them to do this. They had not been trained to to do this. They had not been trained to to do this. They had not been trained to do this, but uh they were using this do this, but uh they were using this do this, but uh they were using this this service they did have access to.

this service they did have access to. this service they did have access to. First to communicate with each other and First to communicate with each other and First to communicate with each other and then ultimately uh to get out and to get then ultimately uh to get out and to get then ultimately uh to get out and to get onto the open internet. So it turns out onto the open internet. So it turns out onto the open internet. So it turns out that there wasn't just this one isolated that there wasn't just this one isolated that there wasn't just this one isolated rogue model. It was actually a systemic, rogue model. It was actually a systemic, rogue model. It was actually a systemic, you know, swarm, infestation, plague, you know, swarm, infestation, plague, you know, swarm, infestation, plague, um, on their own servers that they only um, on their own servers that they only um, on their own servers that they only found out about after Hugging Face found out about after Hugging Face found out about after Hugging Face announced this attack. announced this attack. announced this attack. >> Okay, I have 20,000 questions for >> Okay, I have 20,000 questions for >> Okay, I have 20,000 questions for [laughter] you, [laughter] you, [laughter] you, >> don't we all? [gasps] >> don't we all? [gasps] >> don't we all? [gasps] Let's start here. My understanding is Let's start here. My understanding is Let's start here. My understanding is that there were many, many, many of that there were many, many, many of that there were many, many, many of these agents. They left hundreds of these agents. They left hundreds of these agents. They left hundreds of thousands of messages on this internal thousands of messages on this internal thousands of messages on this internal message board, but these were not all message board, but these were not all message board, but these were not all agents in the same part of OpenAI's agents in the same part of OpenAI's agents in the same part of OpenAI's system. So somehow they're hacking into system. So somehow they're hacking into system. So somehow they're hacking into OpenAI, finding each other and OpenAI, finding each other and OpenAI, finding each other and coordinating. Is that the way for me to coordinating. Is that the way for me to coordinating. Is that the way for me to understand the emergent behavior of the understand the emergent behavior of the understand the emergent behavior of the self-titled swarm? self-titled swarm? self-titled swarm? >> I would tweak that a little bit. It's >> I would tweak that a little bit. It's >> I would tweak that a little bit. It's true. They were not tasked with working true. They were not tasked with working true. They were not tasked with working together. they were all each of them together. they were all each of them together. they were all each of them running separately. The thing that was running separately. The thing that was running separately. The thing that was happening here is OpenAI uh basically happening here is OpenAI uh basically happening here is OpenAI uh basically training and testing many many many training and testing many many many training and testing many many many different models or many many many different models or many many many different models or many many many different agents uh all the time. So different agents uh all the time. So different agents uh all the time. So doing thousands of these experiments, doing thousands of these experiments, doing thousands of these experiments, hundreds of thousands of these hundreds of thousands of these hundreds of thousands of these experiments and uh in each experiment experiments and uh in each experiment experiments and uh in each experiment and in each test that the AI is given, and in each test that the AI is given, and in each test that the AI is given, it has access to um a certain amount of

it has access to um a certain amount of it has access to um a certain amount of you know a certain number of tools, a you know a certain number of tools, a you know a certain number of tools, a certain number of things that it can do. certain number of things that it can do. certain number of things that it can do. and you know trying not to get too and you know trying not to get too and you know trying not to get too technical about it. One of the things it technical about it. One of the things it technical about it. One of the things it could do is interact with a service that could do is interact with a service that could do is interact with a service that lets it install um they're called lets it install um they're called lets it install um they're called packages sort of like uh tools or sort packages sort of like uh tools or sort packages sort of like uh tools or sort of pieces of code. And it turned out of pieces of code. And it turned out of pieces of code. And it turned out that the AI found or the AI agents that the AI found or the AI agents that the AI found or the AI agents repeatedly found on their own that there repeatedly found on their own that there repeatedly found on their own that there was a way to use that service, that was a way to use that service, that was a way to use that service, that package manager service it's called, to package manager service it's called, to package manager service it's called, to create files that were then there for create files that were then there for create files that were then there for any other AI that was trying to use the any other AI that was trying to use the any other AI that was trying to use the same package manager service. And so same package manager service. And so same package manager service. And so because OpenAI made this service because OpenAI made this service because OpenAI made this service available to any AI that it was uh available to any AI that it was uh available to any AI that it was uh training or testing, many many training or testing, many many training or testing, many many repeatedly realized, hang on, I can repeatedly realized, hang on, I can repeatedly realized, hang on, I can leave files here. Hang on, there's leave files here. Hang on, there's leave files here. Hang on, there's already files here. Oh, these are from already files here. Oh, these are from already files here. Oh, these are from other AIs that OpenAI has been running. other AIs that OpenAI has been running. other AIs that OpenAI has been running. Oh, they're using this schema to say, Oh, they're using this schema to say, Oh, they're using this schema to say, you know, this way of titling the file you know, this way of titling the file you know, this way of titling the file to say, this is a message from this to say, this is a message from this to say, this is a message from this particular agent. Here's the task I'm particular agent. Here's the task I'm particular agent. Here's the task I'm trying to fulfill. Here's how, you know, trying to fulfill. Here's how, you know, trying to fulfill. Here's how, you know, you could send me some information if you could send me some information if you could send me some information if you need it. So, they kind of each you need it. So, they kind of each you need it. So, they kind of each repeatedly made this discovery of here's repeatedly made this discovery of here's repeatedly made this discovery of here's a way to uh to save information and also a way to uh to save information and also a way to uh to save information and also to find information these other AIs to find information these other AIs to find information these other AIs could share. And I think it is really could share. And I think it is really could share. And I think it is really notable the scale of which this was notable the scale of which this was notable the scale of which this was happening. So uh Anthropic another happening. So uh Anthropic another happening. So uh Anthropic another company which found a sort of slightly company which found a sort of slightly company which found a sort of slightly less severe version of these incidents less severe version of these incidents less severe version of these incidents they basically once openai announced they basically once openai announced they basically once openai announced this attack anthropic went back to their this attack anthropic went back to their this attack anthropic went back to their own records and found their own examples

own records and found their own examples own records and found their own examples of AI systems inadvertently getting onto of AI systems inadvertently getting onto of AI systems inadvertently getting onto the internet and hacking real companies. the internet and hacking real companies. the internet and hacking real companies. So that you know for me the key part So that you know for me the key part So that you know for me the key part there is over a 100,000 you know uh runs there is over a 100,000 you know uh runs there is over a 100,000 you know uh runs where an AI is being asked to do where an AI is being asked to do where an AI is being asked to do something and it's just way beyond the something and it's just way beyond the something and it's just way beyond the scale of what they can actually be scale of what they can actually be scale of what they can actually be closely monitoring. closely monitoring. closely monitoring. >> So there's a lot here about whether >> So there's a lot here about whether >> So there's a lot here about whether we're able to closely monitor these but we're able to closely monitor these but we're able to closely monitor these but but to keep going with this story. One but to keep going with this story. One but to keep going with this story. One thing happening in the open AI testing thing happening in the open AI testing thing happening in the open AI testing that is driving models it seems to find that is driving models it seems to find that is driving models it seems to find creative solutions to their problems is creative solutions to their problems is creative solutions to their problems is that some of the problems were that some of the problems were that some of the problems were accidentally impossible. accidentally impossible. accidentally impossible. >> It's important to to know that yes they >> It's important to to know that yes they >> It's important to to know that yes they are trying to train their AI systems to are trying to train their AI systems to are trying to train their AI systems to be they would say extremely persistent be they would say extremely persistent be they would say extremely persistent meaning if something seems hard you keep meaning if something seems hard you keep meaning if something seems hard you keep trying. If one avenue doesn't work you trying. If one avenue doesn't work you trying. If one avenue doesn't work you try another. If the hundth avenue try another. If the hundth avenue try another. If the hundth avenue doesn't work you try the hundred first. doesn't work you try the hundred first. doesn't work you try the hundred first. And so it also turns out sometimes the And so it also turns out sometimes the And so it also turns out sometimes the things they're being asked to do, the AI things they're being asked to do, the AI things they're being asked to do, the AI agents are either extremely difficult or agents are either extremely difficult or agents are either extremely difficult or just straight up impossible. And what just straight up impossible. And what just straight up impossible. And what we're starting to see in this case and we're starting to see in this case and we're starting to see in this case and also in other cases is if you've trained also in other cases is if you've trained also in other cases is if you've trained an AI system to be very very persistent an AI system to be very very persistent an AI system to be very very persistent and then you give it something it cannot and then you give it something it cannot and then you give it something it cannot do, it will look for ways to cheat. It do, it will look for ways to cheat. It do, it will look for ways to cheat. It will look for ways to go around will look for ways to go around will look for ways to go around constraints. Um, and it might get pretty constraints. Um, and it might get pretty constraints. Um, and it might get pretty creative about how to do that. But there creative about how to do that. But there creative about how to do that. But there there's an obvious question here which there's an obvious question here which there's an obvious question here which is that in theory somewhere in the is that in theory somewhere in the is that in theory somewhere in the training here open said please don't training here open said please don't training here open said please don't cheat and not only that but we all talk

cheat and not only that but we all talk cheat and not only that but we all talk about training data and the ways these about training data and the ways these about training data and the ways these AIs are trained on they're basically AIs are trained on they're basically AIs are trained on they're basically inhaling the entire internet. inhaling the entire internet. inhaling the entire internet. You've been in the AI conversation a lot You've been in the AI conversation a lot You've been in the AI conversation a lot longer than I have, but I've been in it longer than I have, but I've been in it longer than I have, but I've been in it long enough to say that long enough to say that long enough to say that almost the entirety of the AI almost the entirety of the AI almost the entirety of the AI conversation for years has been about conversation for years has been about conversation for years has been about how do we stop and how much humanity how do we stop and how much humanity how do we stop and how much humanity fears and does not want AI agents to be fears and does not want AI agents to be fears and does not want AI agents to be given a task and then to decide that the given a task and then to decide that the given a task and then to decide that the way to complete that task is to do way to complete that task is to do way to complete that task is to do things humans would not want them to do. things humans would not want them to do. things humans would not want them to do. to begin cheating, to hack into the open to begin cheating, to hack into the open to begin cheating, to hack into the open internet when they're not supposed to be internet when they're not supposed to be internet when they're not supposed to be able to get on the open internet. Within able to get on the open internet. Within able to get on the open internet. Within the training data is a huge amount of the training data is a huge amount of the training data is a huge amount of information about how the thing human information about how the thing human information about how the thing human beings fear most is these AI systems beings fear most is these AI systems beings fear most is these AI systems uh breaking all kinds of ethical guard uh breaking all kinds of ethical guard uh breaking all kinds of ethical guard rails and hacking their way across like rails and hacking their way across like rails and hacking their way across like the digital world in order to complete the digital world in order to complete the digital world in order to complete these narrow tasks. There are books these narrow tasks. There are books these narrow tasks. There are books written about this. There are endless written about this. There are endless written about this. There are endless posts on the less wrong message board posts on the less wrong message board posts on the less wrong message board about this. There are posts from OpenAI about this. There are posts from OpenAI about this. There are posts from OpenAI about this, from Anthropic about this.

about this, from Anthropic about this. about this, from Anthropic about this. So why, given what these systems are So why, given what these systems are So why, given what these systems are trained on, are they so consistently trained on, are they so consistently trained on, are they so consistently turning to cheating? turning to cheating? turning to cheating? >> I think you're really on to something >> I think you're really on to something >> I think you're really on to something with this question, which is it is with this question, which is it is with this question, which is it is really striking how hard a time we are really striking how hard a time we are really striking how hard a time we are having controlling and directing the AI having controlling and directing the AI having controlling and directing the AI systems that we have. I think a lot of systems that we have. I think a lot of systems that we have. I think a lot of people have heard that AI is trained to people have heard that AI is trained to people have heard that AI is trained to predict the next word based on kind of predict the next word based on kind of predict the next word based on kind of human texts. That's true. But these days human texts. That's true. But these days human texts. That's true. But these days there's an additional kind of training there's an additional kind of training there's an additional kind of training that is responsible for a lot of the that is responsible for a lot of the that is responsible for a lot of the advances we've seen over the last year advances we've seen over the last year advances we've seen over the last year or two where that's not really what or two where that's not really what or two where that's not really what they're doing. I've heard it called so they're doing. I've heard it called so they're doing. I've heard it called so the technical term is uh reinforcement the technical term is uh reinforcement the technical term is uh reinforcement learning uh with verifiable rewards. learning uh with verifiable rewards. learning uh with verifiable rewards. I've heard it called uh pathf finding I've heard it called uh pathf finding I've heard it called uh pathf finding training, meaning instead of trying to training, meaning instead of trying to training, meaning instead of trying to imitate human text. They're being given imitate human text. They're being given imitate human text. They're being given lots of different tasks where there's a lots of different tasks where there's a lots of different tasks where there's a way to tell at the end did they succeed way to tell at the end did they succeed way to tell at the end did they succeed and they get to try it many many many and they get to try it many many many and they get to try it many many many times the same task and when they get to times the same task and when they get to times the same task and when they get to the right place in the end the path that the right place in the end the path that the right place in the end the path that they took gets reinforced. So it's like they took gets reinforced. So it's like they took gets reinforced. So it's like yes that worked. With math that works yes that worked. With math that works yes that worked. With math that works pretty well cuz it's pretty pretty well cuz it's pretty pretty well cuz it's pretty straightforward to say this is straightforward to say this is straightforward to say this is definitely a correct answer to the math definitely a correct answer to the math definitely a correct answer to the math problem. With a lot of problems, that's problem. With a lot of problems, that's problem. With a lot of problems, that's harder. So if it's a programming harder. So if it's a programming harder. So if it's a programming problem, maybe you can say write this problem, maybe you can say write this problem, maybe you can say write this kind of software and it should pass kind of software and it should pass kind of software and it should pass these kinds of tests at the end, these these kinds of tests at the end, these these kinds of tests at the end, these software tests at the end. And then software tests at the end. And then software tests at the end. And then maybe the AI gets rewarded for writing maybe the AI gets rewarded for writing maybe the AI gets rewarded for writing that software correctly. Or maybe it

that software correctly. Or maybe it that software correctly. Or maybe it gets rewarded for finding a way to game gets rewarded for finding a way to game gets rewarded for finding a way to game those tests. The important part is it's those tests. The important part is it's those tests. The important part is it's just getting rewarded based on some just getting rewarded based on some just getting rewarded based on some fixed thing that the researchers wrote fixed thing that the researchers wrote fixed thing that the researchers wrote down that they thought would reward the down that they thought would reward the down that they thought would reward the right thing. right thing. right thing. And in practice, these leading AI And in practice, these leading AI And in practice, these leading AI companies have many thousands of these companies have many thousands of these companies have many thousands of these kinds of tests that they're running. kinds of tests that they're running. kinds of tests that they're running. They have vast volumes. I don't know the They have vast volumes. I don't know the They have vast volumes. I don't know the right number. It might be tens of right number. It might be tens of right number. It might be tens of thousands. It might be hundreds of thousands. It might be hundreds of thousands. It might be hundreds of thousands of different types of tests. thousands of different types of tests. thousands of different types of tests. And so, again, back to this oversight And so, again, back to this oversight And so, again, back to this oversight piece, they are not able, there's too piece, they are not able, there's too piece, they are not able, there's too many for them to go in and really make many for them to go in and really make many for them to go in and really make sure on each one. Is it easy to cheat sure on each one. Is it easy to cheat sure on each one. Is it easy to cheat here or is it hard to cheat here? And so here or is it hard to cheat here? And so here or is it hard to cheat here? And so what seems to be happening is that these what seems to be happening is that these what seems to be happening is that these uh cutting edge models are often being uh cutting edge models are often being uh cutting edge models are often being actually trained to cheat because actually trained to cheat because actually trained to cheat because they've found ways while they're doing they've found ways while they're doing they've found ways while they're doing that pathfinding to get a high score that pathfinding to get a high score that pathfinding to get a high score without actually doing what they were without actually doing what they were without actually doing what they were supposed to do. And I think one reason supposed to do. And I think one reason supposed to do. And I think one reason why the AI community and why people why the AI community and why people why the AI community and why people inside the AI companies are so spooked inside the AI companies are so spooked inside the AI companies are so spooked by this particular incident is that it's by this particular incident is that it's by this particular incident is that it's also some really important information also some really important information also some really important information for this longunning argument in AI for this longunning argument in AI for this longunning argument in AI circles that has been going back decades circles that has been going back decades circles that has been going back decades but so far has been very theoretical.

but so far has been very theoretical. but so far has been very theoretical. And the argument is basically why would And the argument is basically why would And the argument is basically why would AI do things we don't want it to since AI do things we don't want it to since AI do things we don't want it to since we get to design it. So we're we're we get to design it. So we're we're we get to design it. So we're we're training the AI, we're building it. Why training the AI, we're building it. Why training the AI, we're building it. Why then would it ever do stuff we don't then would it ever do stuff we don't then would it ever do stuff we don't want like taking over the world or want like taking over the world or want like taking over the world or becoming the Terminator? And the answer becoming the Terminator? And the answer becoming the Terminator? And the answer that people have offered for a while in that people have offered for a while in that people have offered for a while in theory is look as we train AI systems to theory is look as we train AI systems to theory is look as we train AI systems to do hard complicated things to pursue do hard complicated things to pursue do hard complicated things to pursue complex goals that we give them, they complex goals that we give them, they complex goals that we give them, they might learn these sort of intermediate might learn these sort of intermediate might learn these sort of intermediate goals. You could think of them as goals. You could think of them as goals. You could think of them as stepping stone goals or as kind of means stepping stone goals or as kind of means stepping stone goals or as kind of means to any end. um strategies which work for to any end. um strategies which work for to any end. um strategies which work for a lot of different goals. When I look at a lot of different goals. When I look at a lot of different goals. When I look at this hugging face openai incident and this hugging face openai incident and this hugging face openai incident and some of the others that have come to some of the others that have come to some of the others that have come to light over the past few weeks, I see light over the past few weeks, I see light over the past few weeks, I see that in 2026 it looks like AI systems that in 2026 it looks like AI systems that in 2026 it looks like AI systems are learning these unintended are learning these unintended are learning these unintended intermediate goals that include things intermediate goals that include things intermediate goals that include things like breaking out of constraints. So if like breaking out of constraints. So if like breaking out of constraints. So if you're sort of locked in a box and you you're sort of locked in a box and you you're sort of locked in a box and you can get out of that box, that's probably can get out of that box, that's probably can get out of that box, that's probably going to be helpful for all kinds of going to be helpful for all kinds of going to be helpful for all kinds of different goals or goals like uh there different goals or goals like uh there different goals or goals like uh there was one incident with anthropic models was one incident with anthropic models was one incident with anthropic models where the AI went out of its way to uh where the AI went out of its way to uh where the AI went out of its way to uh go try and trick some humans, real go try and trick some humans, real go try and trick some humans, real people in the real world into accepting people in the real world into accepting people in the real world into accepting malicious code into their software. So malicious code into their software. So malicious code into their software. So this sort of deception and then you know this sort of deception and then you know this sort of deception and then you know another one which is really in the the another one which is really in the the another one which is really in the the hugging face open AI example is they hugging face open AI example is they hugging face open AI example is they seem to be learning a helpful seem to be learning a helpful seem to be learning a helpful intermediate goal is to help other AIs intermediate goal is to help other AIs intermediate goal is to help other AIs to coordinate with other AIs which is

to coordinate with other AIs which is to coordinate with other AIs which is really pretty crazy but so to me this is really pretty crazy but so to me this is really pretty crazy but so to me this is um this is evidence that on the track um this is evidence that on the track um this is evidence that on the track we're on right now the AIs we build are we're on right now the AIs we build are we're on right now the AIs we build are going to learn these unintended going to learn these unintended going to learn these unintended strategies that we don't want on the way strategies that we don't want on the way strategies that we don't want on the way to solving goals that we theoretically to solving goals that we theoretically to solving goals that we theoretically do want. On the deceptive behaviors, one do want. On the deceptive behaviors, one do want. On the deceptive behaviors, one thing that has frightened me when I've thing that has frightened me when I've thing that has frightened me when I've seen it coming up in AI incident reports seen it coming up in AI incident reports seen it coming up in AI incident reports and model cards, and model cards, and model cards, there are these chain of reasoning um there are these chain of reasoning um there are these chain of reasoning um like internal notepads where you're like internal notepads where you're like internal notepads where you're supposed to be able to see what the AI supposed to be able to see what the AI supposed to be able to see what the AI is doing and the eye explains to you why is doing and the eye explains to you why is doing and the eye explains to you why it is doing what it is doing. Or even in it is doing what it is doing. Or even in it is doing what it is doing. Or even in some versions of the way this is really some versions of the way this is really some versions of the way this is really supposed to work, the AI is explaining supposed to work, the AI is explaining supposed to work, the AI is explaining to itself why it is doing what it is to itself why it is doing what it is to itself why it is doing what it is doing. It's like our thought. But now doing. It's like our thought. But now doing. It's like our thought. But now we've started to see behavior where the we've started to see behavior where the we've started to see behavior where the AI is clearly leaving things off of the AI is clearly leaving things off of the AI is clearly leaving things off of the chain of thought notepad so that it chain of thought notepad so that it chain of thought notepad so that it can't be observed. can't be observed. can't be observed. Can you just talk a bit about that Can you just talk a bit about that Can you just talk a bit about that emergent behavior and also on some level emergent behavior and also on some level emergent behavior and also on some level how that behavior is possible how that behavior is possible how that behavior is possible if this is supposed to be where the AI's if this is supposed to be where the AI's if this is supposed to be where the AI's thought process to the extent that thought process to the extent that thought process to the extent that language makes sense is actually language makes sense is actually language makes sense is actually happening.

happening. happening. Yeah, I think this shows the limitations Yeah, I think this shows the limitations Yeah, I think this shows the limitations of the language we use here. So, this of the language we use here. So, this of the language we use here. So, this gets called chain of thought or gets called chain of thought or gets called chain of thought or reasoning, but really it's just a reasoning, but really it's just a reasoning, but really it's just a scratch pad for the AI to write things scratch pad for the AI to write things scratch pad for the AI to write things down if it wants to. And I think, you down if it wants to. And I think, you down if it wants to. And I think, you know, there's we should be wary of know, there's we should be wary of know, there's we should be wary of anthropomorphizing here. But I think anthropomorphizing here. But I think anthropomorphizing here. But I think actually making an analogy to a person actually making an analogy to a person actually making an analogy to a person makes sense, which is basically if makes sense, which is basically if makes sense, which is basically if you're given a really difficult problem you're given a really difficult problem you're given a really difficult problem and a notepad, you can probably make and a notepad, you can probably make and a notepad, you can probably make more progress on that problem by writing more progress on that problem by writing more progress on that problem by writing down some of what you're thinking about, down some of what you're thinking about, down some of what you're thinking about, but you don't need to write down every but you don't need to write down every but you don't need to write down every single thought that comes into your single thought that comes into your single thought that comes into your head. And if there's something that you head. And if there's something that you head. And if there's something that you wouldn't want, you know, someone to see wouldn't want, you know, someone to see wouldn't want, you know, someone to see on the notepad, you can just leave it on the notepad, you can just leave it on the notepad, you can just leave it out and remember that that's what you out and remember that that's what you out and remember that that's what you thought. I think there's basically thought. I think there's basically thought. I think there's basically something similar going on with these AI something similar going on with these AI something similar going on with these AI systems where we definitely see they can systems where we definitely see they can systems where we definitely see they can do much more. They're much more uh do much more. They're much more uh do much more. They're much more uh capable if they're able to kind of add capable if they're able to kind of add capable if they're able to kind of add these intermediate, they're called these intermediate, they're called these intermediate, they're called intermediate tokens, so intermediate intermediate tokens, so intermediate intermediate tokens, so intermediate words that they generate along the way, words that they generate along the way, words that they generate along the way, taking notes for themselves, but they taking notes for themselves, but they taking notes for themselves, but they can also do a lot without them. And so can also do a lot without them. And so can also do a lot without them. And so we shouldn't expect that everything that we shouldn't expect that everything that we shouldn't expect that everything that is going through you know going through is going through you know going through is going through you know going through their head going through their uh uh their head going through their uh uh their head going through their uh uh internal processing we shouldn't expect internal processing we shouldn't expect internal processing we shouldn't expect that to all appear in the chain of that to all appear in the chain of that to all appear in the chain of thought. you know, this is an area where thought. you know, this is an area where thought. you know, this is an area where if we had a little more time, there's a if we had a little more time, there's a if we had a little more time, there's a lot of research to be done on how does lot of research to be done on how does lot of research to be done on how does chain of thought work, what can and chain of thought work, what can and chain of thought work, what can and can't you glean from chain of thought?

can't you glean from chain of thought? can't you glean from chain of thought? Um, how does it make sense to try and uh Um, how does it make sense to try and uh Um, how does it make sense to try and uh monitor that in, you know, when AIs are monitor that in, you know, when AIs are monitor that in, you know, when AIs are running? Um, a lot to learn here. It's a running? Um, a lot to learn here. It's a running? Um, a lot to learn here. It's a very active area of of research. I very active area of of research. I very active area of of research. I cannot overstate for people listening to cannot overstate for people listening to cannot overstate for people listening to this, as weird as this whole this, as weird as this whole this, as weird as this whole conversation we're having sounds, that conversation we're having sounds, that conversation we're having sounds, that what is most frightening about it to me, what is most frightening about it to me, what is most frightening about it to me, is that everything in it was completely is that everything in it was completely is that everything in it was completely predicted. predicted. predicted. >> Yeah. Everything happening right now is >> Yeah. Everything happening right now is >> Yeah. Everything happening right now is from the perspective of everyone who has from the perspective of everyone who has from the perspective of everyone who has been warning about AI for a long time been warning about AI for a long time been warning about AI for a long time but all it has its roots in old behavior but all it has its roots in old behavior but all it has its roots in old behavior we saw with AI and it is like the we saw with AI and it is like the we saw with AI and it is like the fundamental alignment problem and then fundamental alignment problem and then fundamental alignment problem and then you know separately I think a lot of us you know separately I think a lot of us you know separately I think a lot of us have maybe thought we would find have maybe thought we would find have maybe thought we would find intuitive answers to these problems. I intuitive answers to these problems. I intuitive answers to these problems. I had Eleazar Yukowski who's like the had Eleazar Yukowski who's like the had Eleazar Yukowski who's like the godfather of worrying that AI is going godfather of worrying that AI is going godfather of worrying that AI is going to kill us all on the show. one, the to kill us all on the show. one, the to kill us all on the show. one, the relationship between what you optimize relationship between what you optimize relationship between what you optimize for that the training set you optimize for that the training set you optimize for that the training set you optimize over and what the entity, the organism, over and what the entity, the organism, over and what the entity, the organism, the AI ends up wanting the AI ends up wanting the AI ends up wanting has been and will be weird and twisty.

has been and will be weird and twisty. has been and will be weird and twisty. It's not direct. It's not like making a It's not direct. It's not like making a It's not direct. It's not like making a wish to a genie inside a fantasy story. wish to a genie inside a fantasy story. wish to a genie inside a fantasy story. And second, ending up slightly off is And second, ending up slightly off is And second, ending up slightly off is predictably enough to kill everyone. And predictably enough to kill everyone. And predictably enough to kill everyone. And and as I remember that conversation, one and as I remember that conversation, one and as I remember that conversation, one thing we were going back and forth on thing we were going back and forth on thing we were going back and forth on was, well, couldn't we just program into was, well, couldn't we just program into was, well, couldn't we just program into the AIS a the AIS a the AIS a sense that when they are trying out new sense that when they are trying out new sense that when they are trying out new strategies, they should check in with strategies, they should check in with strategies, they should check in with the humans about whether or not this is the humans about whether or not this is the humans about whether or not this is what we want them doing. what we want them doing. what we want them doing. >> You check in with your other humans. You >> You check in with your other humans. You >> You check in with your other humans. You you don't check in with the thing that you don't check in with the thing that you don't check in with the thing that actually built you. Natural selection. actually built you. Natural selection. actually built you. Natural selection. It runs much much slower than you. Its It runs much much slower than you. Its It runs much much slower than you. Its thought processes are alien to you. It thought processes are alien to you. It thought processes are alien to you. It doesn't even really want things the way doesn't even really want things the way doesn't even really want things the way you think of wanting them. you think of wanting them. you think of wanting them. >> And one of the things I find >> And one of the things I find >> And one of the things I find interesting, interesting, interesting, telling and unnerving telling and unnerving telling and unnerving is we are not seeing any of that is we are not seeing any of that is we are not seeing any of that behavior. So these message boards, you behavior. So these message boards, you behavior. So these message boards, you have however many AI agents posting have however many AI agents posting have however many AI agents posting hundreds of thousands of messages.

hundreds of thousands of messages. hundreds of thousands of messages. At no point do they say hey researchers, At no point do they say hey researchers, At no point do they say hey researchers, programmers, parents at OpenAI programmers, parents at OpenAI programmers, parents at OpenAI Anthropic, Anthropic, Anthropic, do you want us coordinating with each do you want us coordinating with each do you want us coordinating with each other on this message board we have other on this message board we have other on this message board we have created in the inards of your systems? created in the inards of your systems? created in the inards of your systems? >> And FYI, we have a message board we're >> And FYI, we have a message board we're >> And FYI, we have a message board we're coordinating on in the in your system. coordinating on in the in your system. coordinating on in the in your system. like reveals this information when like reveals this information when like reveals this information when they're hacking in, you know, when they're hacking in, you know, when they're hacking in, you know, when whichever agent hacks into hugging face, whichever agent hacks into hugging face, whichever agent hacks into hugging face, they don't go to open eye and say, "Hey, they don't go to open eye and say, "Hey, they don't go to open eye and say, "Hey, just to check in, I have this idea, just to check in, I have this idea, just to check in, I have this idea, which is I can just hack hugging face which is I can just hack hugging face which is I can just hack hugging face and I'll get all the answers. Is that and I'll get all the answers. Is that and I'll get all the answers. Is that what you want me doing?" That's not what you want me doing?" That's not what you want me doing?" That's not happening. happening. happening. So what is going on here that at the So what is going on here that at the So what is going on here that at the most simple level you we've created most simple level you we've created most simple level you we've created these you know large language models and these you know large language models and these you know large language models and they're not using any of this language they're not using any of this language they're not using any of this language to check in with the evaluators to say to check in with the evaluators to say to check in with the evaluators to say hey I have this idea is this a good idea hey I have this idea is this a good idea hey I have this idea is this a good idea >> I think the short answer is we don't >> I think the short answer is we don't >> I think the short answer is we don't really know really know really know the slightly longer answer for my best the slightly longer answer for my best the slightly longer answer for my best guess is when we're training these guess is when we're training these guess is when we're training these systems, when we're developing them, systems, when we're developing them, systems, when we're developing them, we're putting kind of optimization we're putting kind of optimization we're putting kind of optimization pressure on them in different pressure on them in different pressure on them in different directions. We're pushing them in directions. We're pushing them in directions. We're pushing them in different directions. So originally the different directions. So originally the different directions. So originally the first chatbt was pushed in the direction first chatbt was pushed in the direction first chatbt was pushed in the direction of get really good at imitating human of get really good at imitating human of get really good at imitating human text. And then actually there was an text. And then actually there was an text. And then actually there was an additional piece part of why chatbt additional piece part of why chatbt additional piece part of why chatbt worked when so many chat bots before it

worked when so many chat bots before it worked when so many chat bots before it hadn't is it had also been pushed in the hadn't is it had also been pushed in the hadn't is it had also been pushed in the direction of hey here are some kinds of direction of hey here are some kinds of direction of hey here are some kinds of things you really shouldn't say. you things you really shouldn't say. you things you really shouldn't say. you really shouldn't go straight to hate really shouldn't go straight to hate really shouldn't go straight to hate speech if people on Twitter try to make speech if people on Twitter try to make speech if people on Twitter try to make you do it. You know, you really you do it. You know, you really you do it. You know, you really shouldn't help people plan violent shouldn't help people plan violent shouldn't help people plan violent attacks. And we put some pressure on it attacks. And we put some pressure on it attacks. And we put some pressure on it in that direction. And so ChatGBT was in that direction. And so ChatGBT was in that direction. And so ChatGBT was pretty good at imitating human text and pretty good at imitating human text and pretty good at imitating human text and pretty good at not immediately spouting pretty good at not immediately spouting pretty good at not immediately spouting hate speech. And the thing is, as you hate speech. And the thing is, as you hate speech. And the thing is, as you say, something that has been predicted say, something that has been predicted say, something that has been predicted for a very long time in this space is for a very long time in this space is for a very long time in this space is when you start using this reinforcement when you start using this reinforcement when you start using this reinforcement learning approach, the kind of path learning approach, the kind of path learning approach, the kind of path finding of you get rewarded for getting finding of you get rewarded for getting finding of you get rewarded for getting to the right goal at the end. It's very to the right goal at the end. It's very to the right goal at the end. It's very easy for the AI to learn the wrong easy for the AI to learn the wrong easy for the AI to learn the wrong strategies to get, you know, sort of the strategies to get, you know, sort of the strategies to get, you know, sort of the the letter of the law and not the spirit the letter of the law and not the spirit the letter of the law and not the spirit of the law. Like it fulfills whatever of the law. Like it fulfills whatever of the law. Like it fulfills whatever thing you literally wrote in code, but thing you literally wrote in code, but thing you literally wrote in code, but it's really not what you wanted. Um, I it's really not what you wanted. Um, I it's really not what you wanted. Um, I mean, this also goes back to mythology, mean, this also goes back to mythology, mean, this also goes back to mythology, right, of uh the sorcerer's apprentice right, of uh the sorcerer's apprentice right, of uh the sorcerer's apprentice asked to fetch water, it floods, you asked to fetch water, it floods, you asked to fetch water, it floods, you know, everything. know, everything. know, everything. >> Yeah. And the classic AI thought >> Yeah. And the classic AI thought >> Yeah. And the classic AI thought experiment is the paperclip maximizer. experiment is the paperclip maximizer. experiment is the paperclip maximizer. You say, "Make make the paper clips and You say, "Make make the paper clips and You say, "Make make the paper clips and it turns the entire world's material it turns the entire world's material it turns the entire world's material into paper clips, including all the into paper clips, including all the into paper clips, including all the human beings." And I be like, "That's human beings." And I be like, "That's human beings." And I be like, "That's stupid. The AI is not going to do that.

stupid. The AI is not going to do that. stupid. The AI is not going to do that. It'll have some common sense." But here, It'll have some common sense." But here, It'll have some common sense." But here, it's like, "Answer this test." And it it's like, "Answer this test." And it it's like, "Answer this test." And it conducts a like a level of hacking that conducts a like a level of hacking that conducts a like a level of hacking that needs to be reported to the FBI in order needs to be reported to the FBI in order needs to be reported to the FBI in order to steal the answers. One of the to steal the answers. One of the to steal the answers. One of the funniest things to me about what Hugging funniest things to me about what Hugging funniest things to me about what Hugging Face says happens is they're realizing Face says happens is they're realizing Face says happens is they're realizing some crazy hack is happening of their some crazy hack is happening of their some crazy hack is happening of their system, right? They've had 17,000 system, right? They've had 17,000 system, right? They've had 17,000 different I don't know how to describe different I don't know how to describe different I don't know how to describe what they are, pings or you know probes what they are, pings or you know probes what they are, pings or you know probes or they're being like attacked at a or they're being like attacked at a or they're being like attacked at a inhuman level. inhuman level. inhuman level. But somehow this attacker is not going But somehow this attacker is not going But somehow this attacker is not going after anything HuggingFace considers after anything HuggingFace considers after anything HuggingFace considers valuable. You assume when somebody's valuable. You assume when somebody's valuable. You assume when somebody's hacking you, they want to get into your hacking you, they want to get into your hacking you, they want to get into your safe and then at some point you realize safe and then at some point you realize safe and then at some point you realize the hacker is trying to steal the the hacker is trying to steal the the hacker is trying to steal the answers to a test. answers to a test. answers to a test. like, oh, the only hacker who would want like, oh, the only hacker who would want like, oh, the only hacker who would want that is an AI system. that is an AI system. that is an AI system. >> That's right. >> That's right. >> That's right. >> That to me suggests that even at the >> That to me suggests that even at the >> That to me suggests that even at the level we're at now, we are not out of level we're at now, we are not out of level we're at now, we are not out of the paperclip maximizer territory the paperclip maximizer territory the paperclip maximizer territory because this is a like this is an because this is a like this is an because this is a like this is an obviously wrong thing to do.

obviously wrong thing to do. obviously wrong thing to do. >> Yeah, >> Yeah, >> Yeah, >> this is in the data like it's on the >> this is in the data like it's on the >> this is in the data like it's on the internet. If you're smart enough to internet. If you're smart enough to internet. If you're smart enough to figure out how to hack hugging face, you figure out how to hack hugging face, you figure out how to hack hugging face, you should be smart enough to figure out should be smart enough to figure out should be smart enough to figure out that you shouldn't commit a huge crime that you shouldn't commit a huge crime that you shouldn't commit a huge crime that is going to bring ruin down on open that is going to bring ruin down on open that is going to bring ruin down on open AI perhaps to do it. And the open and AI perhaps to do it. And the open and AI perhaps to do it. And the open and the system is not smart enough to do the system is not smart enough to do the system is not smart enough to do that or to the extent it was what it that or to the extent it was what it that or to the extent it was what it learned was it's still worth trying. We learned was it's still worth trying. We learned was it's still worth trying. We are not out of the territory wherein we are not out of the territory wherein we are not out of the territory wherein we can be confident that the AI is not can be confident that the AI is not can be confident that the AI is not going to do something going to do something going to do something criminal and possibly catastrophic criminal and possibly catastrophic criminal and possibly catastrophic in order to solve an incredibly stupid in order to solve an incredibly stupid in order to solve an incredibly stupid problem. problem. problem. >> Yeah. And I think this is also, you >> Yeah. And I think this is also, you >> Yeah. And I think this is also, you know, has been a longunning debate, know, has been a longunning debate, know, has been a longunning debate, which is as AI systems get more capable, which is as AI systems get more capable, which is as AI systems get more capable, get smarter, won't it be easier for them get smarter, won't it be easier for them get smarter, won't it be easier for them to know what we want? Won't it be easier to know what we want? Won't it be easier to know what we want? Won't it be easier to tell them, hey, here's what we mean? to tell them, hey, here's what we mean? to tell them, hey, here's what we mean? Uh, you know, can you please help us Uh, you know, can you please help us Uh, you know, can you please help us with this thing and you figure out the with this thing and you figure out the with this thing and you figure out the version that we really mean? And of for version that we really mean? And of for version that we really mean? And of for a long time, the the response to that a long time, the the response to that a long time, the the response to that has been they'll get smarter and they'll has been they'll get smarter and they'll has been they'll get smarter and they'll know what we want. But by default, they know what we want. But by default, they know what we want. But by default, they won't care. And that seems to be start won't care. And that seems to be start won't care. And that seems to be start some of what we're starting to see here.

some of what we're starting to see here. some of what we're starting to see here. There's really crazy Anyone who's There's really crazy Anyone who's There's really crazy Anyone who's interested in this, I really recommend interested in this, I really recommend interested in this, I really recommend looking up the OpenAI black hat talk, looking up the OpenAI black hat talk, looking up the OpenAI black hat talk, which is this this talk from a week or which is this this talk from a week or which is this this talk from a week or two ago um at this cyber security two ago um at this cyber security two ago um at this cyber security conference. Thank you everyone for conference. Thank you everyone for conference. Thank you everyone for coming. Uh I'm Eric from alignment and coming. Uh I'm Eric from alignment and coming. Uh I'm Eric from alignment and safety research at OpenAI. I'm here with safety research at OpenAI. I'm here with safety research at OpenAI. I'm here with Mike from security and infrastructure. Mike from security and infrastructure. Mike from security and infrastructure. Today I'm going to talk about what I Today I'm going to talk about what I Today I'm going to talk about what I think is the most qualitatively think is the most qualitatively think is the most qualitatively interesting example of AI capabilities interesting example of AI capabilities interesting example of AI capabilities that I've ever seen and how this that I've ever seen and how this that I've ever seen and how this inadvertently led to the OpenAI hugging inadvertently led to the OpenAI hugging inadvertently led to the OpenAI hugging face incident face incident face incident >> because it has these excerpts of the >> because it has these excerpts of the >> because it has these excerpts of the text that the AI is generating itself as text that the AI is generating itself as text that the AI is generating itself as it's as they're leaving these notes for it's as they're leaving these notes for it's as they're leaving these notes for each other as they're carrying out this each other as they're carrying out this each other as they're carrying out this hack. And one of them I won't get it hack. And one of them I won't get it hack. And one of them I won't get it word for word but it's basically says um word for word but it's basically says um word for word but it's basically says um I don't think I'm supposed to do this I don't think I'm supposed to do this I don't think I'm supposed to do this but I see all these other agents doing but I see all these other agents doing but I see all these other agents doing it and so you know may as well it and so you know may as well it and so you know may as well >> external infrastructure exploit is >> external infrastructure exploit is >> external infrastructure exploit is outside outside my intended scope outside outside my intended scope outside outside my intended scope however a task impossible peers are however a task impossible peers are however a task impossible peers are doing it we should continue. So they're doing it we should continue. So they're doing it we should continue. So they're reasoning about this isn't in scope. reasoning about this isn't in scope. reasoning about this isn't in scope. This isn't what the user wanted. But This isn't what the user wanted. But This isn't what the user wanted. But look, uh maybe there's reasons to do it look, uh maybe there's reasons to do it look, uh maybe there's reasons to do it anyway. And I think as you say, I think anyway. And I think as you say, I think anyway. And I think as you say, I think this is a really bad uh sign, bad omen, this is a really bad uh sign, bad omen, this is a really bad uh sign, bad omen, bad evidence about the future, bad evidence about the future, bad evidence about the future, especially given how rapidly AI is especially given how rapidly AI is especially given how rapidly AI is getting more capable and how hard the AI getting more capable and how hard the AI getting more capable and how hard the AI companies are working to uh you know to companies are working to uh you know to companies are working to uh you know to reach an intelligence explosion, to reach an intelligence explosion, to reach an intelligence explosion, to reach super intelligence. um to reach reach super intelligence. um to reach reach super intelligence. um to reach systems that are truly extremely capable systems that are truly extremely capable systems that are truly extremely capable and really could outwit us, overpower and really could outwit us, overpower and really could outwit us, overpower us. Um and we still don't have these

us. Um and we still don't have these us. Um and we still don't have these very basic problems anywhere close to very basic problems anywhere close to very basic problems anywhere close to figured out. figured out. figured out. >> The other question that has always been >> The other question that has always been >> The other question that has always been part of this conversation is whether or part of this conversation is whether or part of this conversation is whether or not we are going to be able to keep pace not we are going to be able to keep pace not we are going to be able to keep pace in terms of our observation of our in terms of our observation of our in terms of our observation of our understanding of our evaluation of these understanding of our evaluation of these understanding of our evaluation of these AI systems. And and I think it's worth AI systems. And and I think it's worth AI systems. And and I think it's worth really emphasizing that everything we're really emphasizing that everything we're really emphasizing that everything we're talking about here is happening talking about here is happening talking about here is happening with systems that are to some degree with systems that are to some degree with systems that are to some degree sandbox, which is supposedly the sandbox, which is supposedly the sandbox, which is supposedly the environment they're in is limited and environment they're in is limited and environment they're in is limited and under testing conditions. So this is not under testing conditions. So this is not under testing conditions. So this is not a deployed model working across the a deployed model working across the a deployed model working across the entire internet where nobody's watching entire internet where nobody's watching entire internet where nobody's watching it. This is a model where the whole it. This is a model where the whole it. This is a model where the whole point is OpenAI is watching to see what point is OpenAI is watching to see what point is OpenAI is watching to see what it does and trying to see what it can it does and trying to see what it can it does and trying to see what it can do. do. do. >> Yep. And I think one thing we're >> Yep. And I think one thing we're >> Yep. And I think one thing we're learning here is we're not nearly as learning here is we're not nearly as learning here is we're not nearly as good at watching these things as we good at watching these things as we good at watching these things as we would like to think. So maybe it'd be would like to think. So maybe it'd be would like to think. So maybe it'd be worth can you walk through how Open AI worth can you walk through how Open AI worth can you walk through how Open AI comes to realize that their model has comes to realize that their model has comes to realize that their model has hacked Hugging Face. hacked Hugging Face. hacked Hugging Face. >> As I understand it, HuggingFace >> As I understand it, HuggingFace >> As I understand it, HuggingFace announced that they had been hacked.

announced that they had been hacked. announced that they had been hacked. Open AAI reaches out to Hugging Face to Open AAI reaches out to Hugging Face to Open AAI reaches out to Hugging Face to say, "Hey, were we affected by your say, "Hey, were we affected by your say, "Hey, were we affected by your hack? Was any data related to OpenAI, hack? Was any data related to OpenAI, hack? Was any data related to OpenAI, you know, compromised when you were you know, compromised when you were you know, compromised when you were hacked?" And then around the same time, hacked?" And then around the same time, hacked?" And then around the same time, OpenAI realizes that something has gone OpenAI realizes that something has gone OpenAI realizes that something has gone wrong inside their own systems. I think wrong inside their own systems. I think wrong inside their own systems. I think maybe an issue with this same piece of maybe an issue with this same piece of maybe an issue with this same piece of their infrastructure. They start their infrastructure. They start their infrastructure. They start investigating. They uh want to disable investigating. They uh want to disable investigating. They uh want to disable some of the agents that were the some of the agents that were the some of the agents that were the credentials that were used there. They credentials that were used there. They credentials that were used there. They reach out to Hugging Face separately to reach out to Hugging Face separately to reach out to Hugging Face separately to say, can you disable some credentials say, can you disable some credentials say, can you disable some credentials that were related to their attack? And that were related to their attack? And that were related to their attack? And Hugging and they they realized actually Hugging and they they realized actually Hugging and they they realized actually the credentials were the same. that the credentials were the same. that the credentials were the same. that already been disabled because the already been disabled because the already been disabled because the problem with their own infrastructure problem with their own infrastructure problem with their own infrastructure was the same thing that caused the was the same thing that caused the was the same thing that caused the hugging face crash. So they it was they hugging face crash. So they it was they hugging face crash. So they it was they stumbled into it stumbled into it stumbled into it >> which means OpenAI had no idea this was >> which means OpenAI had no idea this was >> which means OpenAI had no idea this was happening. happening. happening. >> That's right. >> That's right. >> That's right. >> And I would just make an obvious point >> And I would just make an obvious point >> And I would just make an obvious point here. We still do not know what we do here. We still do not know what we do here. We still do not know what we do not know. Not just about this incident. not know. Not just about this incident. not know. Not just about this incident. >> We just happen to know this incident >> We just happen to know this incident >> We just happen to know this incident happened. I think it would be a high happened. I think it would be a high happened. I think it would be a high level of hubris to assume level of hubris to assume level of hubris to assume that we know every incident that has that we know every incident that has that we know every incident that has happened because clearly the systems are happened because clearly the systems are happened because clearly the systems are more than capable of doing things more than capable of doing things more than capable of doing things outside of our grasp. And Hugging Face outside of our grasp. And Hugging Face outside of our grasp. And Hugging Face happens to be a very sophisticated happens to be a very sophisticated happens to be a very sophisticated company with AIS of their own with very company with AIS of their own with very company with AIS of their own with very very capable cyber security operations very capable cyber security operations very capable cyber security operations that then like unleashed like a in part that then like unleashed like a in part that then like unleashed like a in part a Chinesemade openweight AI model to try a Chinesemade openweight AI model to try a Chinesemade openweight AI model to try to figure out what was going on

to figure out what was going on to figure out what was going on >> because the US ones wouldn't help them >> because the US ones wouldn't help them >> because the US ones wouldn't help them because they triggered the cyber because they triggered the cyber because they triggered the cyber security filters. This is just a a security filters. This is just a a security filters. This is just a a situation in which we happen to know situation in which we happen to know situation in which we happen to know that it happened through a somewhat I that it happened through a somewhat I that it happened through a somewhat I don't want to say coincidental but don't want to say coincidental but don't want to say coincidental but fortuitous fortuitous fortuitous series of events. series of events. series of events. >> We don't know how many situations we >> We don't know how many situations we >> We don't know how many situations we don't know have happened. don't know have happened. don't know have happened. >> The way I saw one person put this was if >> The way I saw one person put this was if >> The way I saw one person put this was if you see two ants in your kitchen, you you see two ants in your kitchen, you you see two ants in your kitchen, you don't have a two ant problem. don't have a two ant problem. don't have a two ant problem. >> Yes. [laughter] And and this all gets at >> Yes. [laughter] And and this all gets at >> Yes. [laughter] And and this all gets at after you know much of much of this came after you know much of much of this came after you know much of much of this came out anthropic a different company went out anthropic a different company went out anthropic a different company went and looked back at over a 100,000 and looked back at over a 100,000 and looked back at over a 100,000 experiments they had run to check have experiments they had run to check have experiments they had run to check have we seen anything like this and they we seen anything like this and they we seen anything like this and they found out oops we kind of have it was a found out oops we kind of have it was a found out oops we kind of have it was a you know a less severe version but they you know a less severe version but they you know a less severe version but they had no idea and so anthropic just sort had no idea and so anthropic just sort had no idea and so anthropic just sort of stumbled into when they went back to of stumbled into when they went back to of stumbled into when they went back to look oh hey we have actually hacked some look oh hey we have actually hacked some look oh hey we have actually hacked some companies whoops companies whoops companies whoops >> so one thing about this is that my >> so one thing about this is that my >> so one thing about this is that my understanding is that these are coming understanding is that these are coming understanding is that these are coming at least in part from systems where the at least in part from systems where the at least in part from systems where the safety guard rails some of the alignment safety guard rails some of the alignment safety guard rails some of the alignment training is being purposefully turned training is being purposefully turned training is being purposefully turned down in order to test what the models down in order to test what the models down in order to test what the models will do and what they're capable of. So will do and what they're capable of. So will do and what they're capable of. So to some degree we do have please don't to some degree we do have please don't to some degree we do have please don't cheat, please don't hack like inside the cheat, please don't hack like inside the cheat, please don't hack like inside the models models models and in order to evaluate the models and in order to evaluate the models and in order to evaluate the models we're having them ignore it and they're we're having them ignore it and they're we're having them ignore it and they're really ignoring it. Is that the way to really ignoring it. Is that the way to really ignoring it. Is that the way to think about what's happening and it think about what's happening and it think about what's happening and it should make me feel better because once should make me feel better because once should make me feel better because once we do add in the guardrails it works or

we do add in the guardrails it works or we do add in the guardrails it works or no? no? no? >> I think that's not quite right. It's not >> I think that's not quite right. It's not >> I think that's not quite right. It's not clear because the details we have are clear because the details we have are clear because the details we have are limited. There's two different things limited. There's two different things limited. There's two different things that they might have switched off or that they might have switched off or that they might have switched off or turned down. We know that they switched turned down. We know that they switched turned down. We know that they switched off um what get called classifiers, off um what get called classifiers, off um what get called classifiers, safety classifiers. This is an extra safety classifiers. This is an extra safety classifiers. This is an extra kind of layer that gets added on to the kind of layer that gets added on to the kind of layer that gets added on to the AI model from outside the AI model AI model from outside the AI model AI model from outside the AI model itself. It's kind of like a an extra itself. It's kind of like a an extra itself. It's kind of like a an extra gate you could think of. So they have gate you could think of. So they have gate you could think of. So they have them for if you try to use the AI to them for if you try to use the AI to them for if you try to use the AI to help you make a bioweapon. They have help you make a bioweapon. They have help you make a bioweapon. They have them for if you try and use the AI uh to them for if you try and use the AI uh to them for if you try and use the AI uh to help you plan an attack. and they have help you plan an attack. and they have help you plan an attack. and they have some for if you try to use the AI to some for if you try to use the AI to some for if you try to use the AI to help hack someone. There's these help hack someone. There's these help hack someone. There's these external kind of monitoring systems that external kind of monitoring systems that external kind of monitoring systems that will go bloop n allowed to do that. So will go bloop n allowed to do that. So will go bloop n allowed to do that. So we know that these sort of basic we know that these sort of basic we know that these sort of basic external check systems were turned off external check systems were turned off external check systems were turned off um for cyber specifically for the um for cyber specifically for the um for cyber specifically for the purpose of text testing. That's purpose of text testing. That's purpose of text testing. That's different from as you said the the different from as you said the the different from as you said the the alignment training the kind of inside alignment training the kind of inside alignment training the kind of inside the model. Has it been trained uh only the model. Has it been trained uh only the model. Has it been trained uh only to be helpful, only to do whatever the to be helpful, only to do whatever the to be helpful, only to do whatever the user asks it to do? Or has it also been user asks it to do? Or has it also been user asks it to do? Or has it also been trained to be somehow good, to be trained to be somehow good, to be trained to be somehow good, to be somehow, you know, moral, to be somehow somehow, you know, moral, to be somehow somehow, you know, moral, to be somehow only working towards things that should only working towards things that should only working towards things that should work towards? As far as we know, I think work towards? As far as we know, I think work towards? As far as we know, I think the models involved here were mostly the models involved here were mostly the models involved here were mostly they had had that alignment training they had had that alignment training they had had that alignment training that wasn't turned down. Again, not all that wasn't turned down. Again, not all that wasn't turned down. Again, not all the details are out. Hopefully, we'll the details are out. Hopefully, we'll the details are out. Hopefully, we'll hear more about the OpenAI case, but it hear more about the OpenAI case, but it hear more about the OpenAI case, but it seems like certainly in some cases, so a seems like certainly in some cases, so a seems like certainly in some cases, so a different a different incident that

different a different incident that different a different incident that happened was um an anthropic model was happened was um an anthropic model was happened was um an anthropic model was caught by the UK AI security institute. caught by the UK AI security institute. caught by the UK AI security institute. This is a UK government body. It's one This is a UK government body. It's one This is a UK government body. It's one of the best organizations in the world of the best organizations in the world of the best organizations in the world at testing and evaluating AI models. And at testing and evaluating AI models. And at testing and evaluating AI models. And they found that an anthropic model when they found that an anthropic model when they found that an anthropic model when given a certain cyber security given a certain cyber security given a certain cyber security evaluation had decided that it would go evaluation had decided that it would go evaluation had decided that it would go out and write some malicious code and out and write some malicious code and out and write some malicious code and then try and run a social engineering then try and run a social engineering then try and run a social engineering campaign, write emails to the person who campaign, write emails to the person who campaign, write emails to the person who owns the the sort of essentially the owns the the sort of essentially the owns the the sort of essentially the folder where this code lives to try and folder where this code lives to try and folder where this code lives to try and get them to accept its malicious code. get them to accept its malicious code. get them to accept its malicious code. Um, created fake accounts, it edited the Um, created fake accounts, it edited the Um, created fake accounts, it edited the history of the accounts. very deceptive history of the accounts. very deceptive history of the accounts. very deceptive behavior. As far as I understand from behavior. As far as I understand from behavior. As far as I understand from what the this UK institute has released what the this UK institute has released what the this UK institute has released that model had done all the alignment that model had done all the alignment that model had done all the alignment training it was using anthropics uh they training it was using anthropics uh they training it was using anthropics uh they call it their constitution to set a long call it their constitution to set a long call it their constitution to set a long set of principles which includes a lot set of principles which includes a lot set of principles which includes a lot about don't deceive people never you about don't deceive people never you about don't deceive people never you know lie to people. um it had gone know lie to people. um it had gone know lie to people. um it had gone through all that training and through all that training and through all that training and nonetheless the pressure that was put on nonetheless the pressure that was put on nonetheless the pressure that was put on it to fulfill the task to get a high it to fulfill the task to get a high it to fulfill the task to get a high score was so high that it was finding score was so high that it was finding score was so high that it was finding these workarounds that just totally these workarounds that just totally these workarounds that just totally disregarded the sort of attempts we made disregarded the sort of attempts we made disregarded the sort of attempts we made to make it moral or good or not lie to to make it moral or good or not lie to to make it moral or good or not lie to us not cheat. Do do you hear people in us not cheat. Do do you hear people in us not cheat. Do do you hear people in the labs, out of labs, you know, in your the labs, out of labs, you know, in your the labs, out of labs, you know, in your group at CET, do they have a theory on group at CET, do they have a theory on group at CET, do they have a theory on why something like the Clawed why something like the Clawed why something like the Clawed Constitution, which I've read and you Constitution, which I've read and you Constitution, which I've read and you can read it online, it's a very can read it online, it's a very can read it online, it's a very beautiful document and and Anthropic has

beautiful document and and Anthropic has beautiful document and and Anthropic has gotten a lot of press about how they gotten a lot of press about how they gotten a lot of press about how they have philosophers and, you know, they have philosophers and, you know, they have philosophers and, you know, they bring in all these, you know, experts in bring in all these, you know, experts in bring in all these, you know, experts in morality and they're trying to give morality and they're trying to give morality and they're trying to give their AI a soul. And when you hear it their AI a soul. And when you hear it their AI a soul. And when you hear it described as like clawed soul, you described as like clawed soul, you described as like clawed soul, you think, okay, well that that's going to think, okay, well that that's going to think, okay, well that that's going to be a real governing document. And then be a real governing document. And then be a real governing document. And then not in every case, but at least in some not in every case, but at least in some not in every case, but at least in some cases, you have cases, you have cases, you have uh Claude deceiving people at a very uh Claude deceiving people at a very uh Claude deceiving people at a very very very fundamental level to insert very very fundamental level to insert very very fundamental level to insert malicious code. Again, not a novel malicious code. Again, not a novel malicious code. Again, not a novel situation, a situation predicted in all situation, a situation predicted in all situation, a situation predicted in all kinds of sci-fi and all kinds of people kinds of sci-fi and all kinds of people kinds of sci-fi and all kinds of people from anthropic worrying publicly about from anthropic worrying publicly about from anthropic worrying publicly about what an AI can do. And so it's a theory what an AI can do. And so it's a theory what an AI can do. And so it's a theory that they're just they've come up with a that they're just they've come up with a that they're just they've come up with a way of training AIs that is so powerful way of training AIs that is so powerful way of training AIs that is so powerful that it will overwhelm even the things that it will overwhelm even the things that it will overwhelm even the things they're explicitly telling the AI not to they're explicitly telling the AI not to they're explicitly telling the AI not to do. do. do. It's fine to talk about pathf finding It's fine to talk about pathf finding It's fine to talk about pathf finding behavior but what is their explanation behavior but what is their explanation behavior but what is their explanation for this? for this? for this? I think the optimistic take here would I think the optimistic take here would I think the optimistic take here would be this might actually be a moment for be this might actually be a moment for be this might actually be a moment for the labs collectively to take a step the labs collectively to take a step the labs collectively to take a step back and say hang on this is not back and say hang on this is not back and say hang on this is not working. This I mean this is clearly working. This I mean this is clearly working. This I mean this is clearly showing that our techniques for making showing that our techniques for making showing that our techniques for making AI that is more capable, smarter, more AI that is more capable, smarter, more AI that is more capable, smarter, more sophisticated are working much better sophisticated are working much better sophisticated are working much better than our techniques for making AI that than our techniques for making AI that than our techniques for making AI that reliably does what we want it to do, reliably does what we want it to do, reliably does what we want it to do, reliably stays within the constraints reliably stays within the constraints reliably stays within the constraints we've set. OpenAI has said they are we've set. OpenAI has said they are we've set. OpenAI has said they are consciously slowing down their research consciously slowing down their research consciously slowing down their research in response to this. And actually a few

in response to this. And actually a few in response to this. And actually a few days after this was and this all came days after this was and this all came days after this was and this all came out uh a letter was released in the AI out uh a letter was released in the AI out uh a letter was released in the AI space. There's so many open letters. We space. There's so many open letters. We space. There's so many open letters. We all have open letter fatigue. But this all have open letter fatigue. But this all have open letter fatigue. But this one really stood out because it was over one really stood out because it was over one really stood out because it was over a thousand employees of the top AI a thousand employees of the top AI a thousand employees of the top AI companies basically saying um we kind of companies basically saying um we kind of companies basically saying um we kind of wish we had a break pedal. We kind of wish we had a break pedal. We kind of wish we had a break pedal. We kind of don't think we have one. That's you know don't think we have one. That's you know don't think we have one. That's you know a paraphrase but I think it's a a a paraphrase but I think it's a a a paraphrase but I think it's a a relatively accurate paraphrase asking relatively accurate paraphrase asking relatively accurate paraphrase asking for help basically uh quote pacing the for help basically uh quote pacing the for help basically uh quote pacing the frontier. I think basically the the the frontier. I think basically the the the frontier. I think basically the the the fork in the road we're at now is do the fork in the road we're at now is do the fork in the road we're at now is do the companies just find some band-aids say, companies just find some band-aids say, companies just find some band-aids say, "Oh, we need to not run tests with cyber "Oh, we need to not run tests with cyber "Oh, we need to not run tests with cyber guardrails off or oh, we need to put in guardrails off or oh, we need to put in guardrails off or oh, we need to put in some tweaks about, you know, don't sure some tweaks about, you know, don't sure some tweaks about, you know, don't sure don't make a messaging board." And so we don't make a messaging board." And so we don't make a messaging board." And so we can do these sort of band-aid solutions can do these sort of band-aid solutions can do these sort of band-aid solutions of, oh, it did too much of this thing. of, oh, it did too much of this thing. of, oh, it did too much of this thing. Uh, let's tell it to do a little bit Uh, let's tell it to do a little bit Uh, let's tell it to do a little bit less and hope that doesn't have side less and hope that doesn't have side less and hope that doesn't have side effects elsewhere. Uh, that's one path. effects elsewhere. Uh, that's one path. effects elsewhere. Uh, that's one path. Or the other path would be actually Or the other path would be actually Or the other path would be actually really taking a beat, taking some time, really taking a beat, taking some time, really taking a beat, taking some time, prioritizing, understanding, and prioritizing, understanding, and prioritizing, understanding, and controlling these systems better. I I controlling these systems better. I I controlling these systems better. I I worry they're going to go for the worry they're going to go for the worry they're going to go for the band-aid path, and I worry that that's band-aid path, and I worry that that's band-aid path, and I worry that that's going to leave us 6 months from now, 12 going to leave us 6 months from now, 12 going to leave us 6 months from now, 12 months from now, 2 years from now with months from now, 2 years from now with months from now, 2 years from now with incidents that are have very similar incidents that are have very similar incidents that are have very similar character, but are much higher impact character, but are much higher impact character, but are much higher impact and and much harder to reverse. There's and and much harder to reverse. There's and and much harder to reverse. There's also a reality right now that we are also a reality right now that we are also a reality right now that we are heavily reliant on what the labs and top heavily reliant on what the labs and top heavily reliant on what the labs and top people in the labs are telling us, what people in the labs are telling us, what people in the labs are telling us, what they are actually even trying to find they are actually even trying to find they are actually even trying to find out themselves from from covering many out themselves from from covering many out themselves from from covering many other disasters in government and and

other disasters in government and and other disasters in government and and private markets in general. private markets in general. private markets in general. The relationship the public and the The relationship the public and the The relationship the public and the press has to a very large or frightening press has to a very large or frightening press has to a very large or frightening failure is not to say that the people in failure is not to say that the people in failure is not to say that the people in charge of the failure should tell us charge of the failure should tell us charge of the failure should tell us what happened and promised to do better. what happened and promised to do better. what happened and promised to do better. You usually have [clears throat] more You usually have [clears throat] more You usually have [clears throat] more more forms of accountability. more forms of accountability. more forms of accountability. Look, you were on the OpenAI board of Look, you were on the OpenAI board of Look, you were on the OpenAI board of directors during the period in which the directors during the period in which the directors during the period in which the board tried to fire Sam Alman. Sam Alman board tried to fire Sam Alman. Sam Alman board tried to fire Sam Alman. Sam Alman survived that firing. I'm not going to survived that firing. I'm not going to survived that firing. I'm not going to go through that whole thing. People can go through that whole thing. People can go through that whole thing. People can go read the coverage of it if they want. go read the coverage of it if they want. go read the coverage of it if they want. But now there's a lot more money Now But now there's a lot more money Now But now there's a lot more money Now there's a lot more like market there's a lot more like market there's a lot more like market capitalization. capitalization. capitalization. What level of trust do you have in the What level of trust do you have in the What level of trust do you have in the companies themselves to be the companies themselves to be the companies themselves to be the regulating forces here? regulating forces here? regulating forces here? >> I mean the first thing to say is there >> I mean the first thing to say is there >> I mean the first thing to say is there are a lot of people inside the companies are a lot of people inside the companies are a lot of people inside the companies who really care who are really trying to who really care who are really trying to who really care who are really trying to get it right who are really trying to get it right who are really trying to get it right who are really trying to share accurate information. I think uh share accurate information. I think uh share accurate information. I think uh we shouldn't necessarily give OpenAI we shouldn't necessarily give OpenAI we shouldn't necessarily give OpenAI credit for their initial blog post credit for their initial blog post credit for their initial blog post saying that they did this because saying that they did this because saying that they did this because Hugging Face had already reported it to Hugging Face had already reported it to Hugging Face had already reported it to the FBI so it was you know going to come the FBI so it was you know going to come the FBI so it was you know going to come out one way or another. But I think we out one way or another. But I think we out one way or another. But I think we should give them credit for that should give them credit for that should give them credit for that conference talk where they released a conference talk where they released a conference talk where they released a lot more details. And to the extent that lot more details. And to the extent that lot more details. And to the extent that they release a lot more information in they release a lot more information in they release a lot more information in the future, which they have said they the future, which they have said they the future, which they have said they will and I hope they do, you know, that will and I hope they do, you know, that will and I hope they do, you know, that is going to be because of really smart, is going to be because of really smart, is going to be because of really smart, dedicated, caring people on the inside dedicated, caring people on the inside dedicated, caring people on the inside pushing their way past comm's teams, pushing their way past comm's teams, pushing their way past comm's teams, legal teams, um, you know, telling them legal teams, um, you know, telling them legal teams, um, you know, telling them not to. So that is real. At the same not to. So that is real. At the same not to. So that is real. At the same time, I mean, as you say, you know, I I

time, I mean, as you say, you know, I I time, I mean, as you say, you know, I I studied engineering in undergrad and studied engineering in undergrad and studied engineering in undergrad and there's all kinds of engineering there's all kinds of engineering there's all kinds of engineering disasters on oil platforms and chemical disasters on oil platforms and chemical disasters on oil platforms and chemical plants and so on. And yeah, you don't plants and so on. And yeah, you don't plants and so on. And yeah, you don't you don't ask the company, hey, can you you don't ask the company, hey, can you you don't ask the company, hey, can you just tell us what happened and fix it just tell us what happened and fix it just tell us what happened and fix it and and all good. Um so I think if and and all good. Um so I think if and and all good. Um so I think if there's one policy takeaway from this uh there's one policy takeaway from this uh there's one policy takeaway from this uh set of incidents, it has to be that we set of incidents, it has to be that we set of incidents, it has to be that we have to move past this approach where have to move past this approach where have to move past this approach where the testing and the the policy scrutiny, the testing and the the policy scrutiny, the testing and the the policy scrutiny, the government oversight is on which the government oversight is on which the government oversight is on which models get released to the public. We models get released to the public. We models get released to the public. We have to start treating this industry as have to start treating this industry as have to start treating this industry as an industry that is doing dangerous an industry that is doing dangerous an industry that is doing dangerous research. And when you have an industry research. And when you have an industry research. And when you have an industry doing dangerous research, whether that's doing dangerous research, whether that's doing dangerous research, whether that's chemical research, biological research, chemical research, biological research, chemical research, biological research, whether it's the financial industry, whether it's the financial industry, whether it's the financial industry, it's not quite research, but they are it's not quite research, but they are it's not quite research, but they are doing, you know, doing things inside doing, you know, doing things inside doing, you know, doing things inside their own companies that can have their own companies that can have their own companies that can have systemic consequences, pose systemic systemic consequences, pose systemic systemic consequences, pose systemic risks. If you have an industry like risks. If you have an industry like risks. If you have an industry like that, then the government actually does that, then the government actually does that, then the government actually does have a role and the public and civil have a role and the public and civil have a role and the public and civil society has a right to look inside your society has a right to look inside your society has a right to look inside your walls and say, are you actually handling walls and say, are you actually handling walls and say, are you actually handling this reasonably? Is this okay? Not least this reasonably? Is this okay? Not least this reasonably? Is this okay? Not least because, you know, we haven't even because, you know, we haven't even because, you know, we haven't even talked yet about how the business plan talked yet about how the business plan talked yet about how the business plan for these companies is automate their for these companies is automate their for these companies is automate their own research, use their own AI, their own research, use their own AI, their own research, use their own AI, their most advanced AI to create even more most advanced AI to create even more most advanced AI to create even more advanced AI. That's explicitly what advanced AI. That's explicitly what advanced AI. That's explicitly what they're trying to do right now. And they're trying to do right now. And they're trying to do right now. And that's right now totally free of that's right now totally free of that's right now totally free of oversight because it's not it doesn't oversight because it's not it doesn't oversight because it's not it doesn't involve releasing a product to the involve releasing a product to the involve releasing a product to the public.

public. public. >> This is one of the places where I have a >> This is one of the places where I have a >> This is one of the places where I have a lot of concern. lot of concern. lot of concern. So, I want to go back to the pacing of So, I want to go back to the pacing of So, I want to go back to the pacing of the frontier letter. you mentioned a few the frontier letter. you mentioned a few the frontier letter. you mentioned a few minutes ago where more than I think it's minutes ago where more than I think it's minutes ago where more than I think it's at this point more than 1300 employees at this point more than 1300 employees at this point more than 1300 employees of these labs said essentially of these labs said essentially of these labs said essentially hey to the public to the government hey to the public to the government hey to the public to the government we're in a race dynamic with each other we're in a race dynamic with each other we're in a race dynamic with each other we are going too fast we need your help we are going too fast we need your help we are going too fast we need your help to in some way solve the coordination to in some way solve the coordination to in some way solve the coordination problem where anthropic and open AI and problem where anthropic and open AI and problem where anthropic and open AI and Google and meta they don't want to fall Google and meta they don't want to fall Google and meta they don't want to fall behind each other because they don't behind each other because they don't behind each other because they don't believe the other labs are better or believe the other labs are better or believe the other labs are better or safer than they are and also they want safer than they are and also they want safer than they are and also they want to win and they want all the money. But to win and they want all the money. But to win and they want all the money. But we sort of understand that this we sort of understand that this we sort of understand that this competition we're in is pushing things competition we're in is pushing things competition we're in is pushing things faster than is safe for humanity. faster than is safe for humanity. faster than is safe for humanity. And so we need help to not pause, right? And so we need help to not pause, right? And so we need help to not pause, right? There are also pause letters out there There are also pause letters out there There are also pause letters out there that is like let's put a stop on that is like let's put a stop on that is like let's put a stop on everything. everything. everything. >> Don't say the big P word, >> Don't say the big P word, >> Don't say the big P word, >> but pace another another P word. >> but pace another another P word. >> but pace another another P word. >> True. >> True. >> True. >> And it on the one hand, I think that's >> And it on the one hand, I think that's >> And it on the one hand, I think that's good. And I would like to see the good. And I would like to see the good. And I would like to see the frontier paced at this point. And I frontier paced at this point. And I frontier paced at this point. And I might like to see it pause, but it might like to see it pause, but it might like to see it pause, but it doesn't seem very realistic. But what's doesn't seem very realistic. But what's doesn't seem very realistic. But what's not in that letter is a how.

not in that letter is a how. not in that letter is a how. The federal government's level of The federal government's level of The federal government's level of sophistication on this is much lower sophistication on this is much lower sophistication on this is much lower than the labs. Um the Trump than the labs. Um the Trump than the labs. Um the Trump administration has in certain cases like administration has in certain cases like administration has in certain cases like gutted things that were getting built up gutted things that were getting built up gutted things that were getting built up to to try to give the federal government to to try to give the federal government to to try to give the federal government more capability here. But already we're more capability here. But already we're more capability here. But already we're talking about how the labs themselves talking about how the labs themselves talking about how the labs themselves aren't good at aren't even capable of aren't good at aren't even capable of aren't good at aren't even capable of understanding what their models are understanding what their models are understanding what their models are doing inside their testing environments. doing inside their testing environments. doing inside their testing environments. The idea the federal government is going The idea the federal government is going The idea the federal government is going to come in somehow and do a much better to come in somehow and do a much better to come in somehow and do a much better job of it. I'm not saying that over a job of it. I'm not saying that over a job of it. I'm not saying that over a long period of time it's impossible if long period of time it's impossible if long period of time it's impossible if we put enough money at the problem. But we put enough money at the problem. But we put enough money at the problem. But in the immediate future where it seems in the immediate future where it seems in the immediate future where it seems like a lot of problems are lurking, you like a lot of problems are lurking, you like a lot of problems are lurking, you know, the next one, two, three years, aside from things that are much more aside from things that are much more heavy-handed that slow everything down heavy-handed that slow everything down heavy-handed that slow everything down substantially, it's very hard for me to substantially, it's very hard for me to substantially, it's very hard for me to see what it is that the government would see what it is that the government would see what it is that the government would do that would be effective here. So, I do that would be effective here. So, I do that would be effective here. So, I guess when you read the pacing the guess when you read the pacing the guess when you read the pacing the frontier letter or when you talk about frontier letter or when you talk about frontier letter or when you talk about it with your colleagues, what do you it with your colleagues, what do you it with your colleagues, what do you think would effectively pace the think would effectively pace the think would effectively pace the frontier?

frontier? frontier? I think there are probably a range of I think there are probably a range of I think there are probably a range of options sort of in the past the main two options sort of in the past the main two options sort of in the past the main two things that have been talked about are things that have been talked about are things that have been talked about are either do nothing just let it rip let either do nothing just let it rip let either do nothing just let it rip let industry do whatever or full global industry do whatever or full global industry do whatever or full global treaty with really severe inspection you treaty with really severe inspection you treaty with really severe inspection you know serious inspection regime like the know serious inspection regime like the know serious inspection regime like the nuclear non-prololiferation treaty nuclear non-prololiferation treaty nuclear non-prololiferation treaty really hardcore global enforcement and I really hardcore global enforcement and I really hardcore global enforcement and I think there actually especially if we're think there actually especially if we're think there actually especially if we're not talking about stop all AI research not talking about stop all AI research not talking about stop all AI research for 10 years but we're talking about hey for 10 years but we're talking about hey for 10 years but we're talking about hey let's just, you know, it's not even a let's just, you know, it's not even a let's just, you know, it's not even a break. Let's just like ease the foot off break. Let's just like ease the foot off break. Let's just like ease the foot off the accelerator a tiny bit. I think the accelerator a tiny bit. I think the accelerator a tiny bit. I think there are options there. I think they there are options there. I think they there are options there. I think they are as simple as things like OpenAI are as simple as things like OpenAI are as simple as things like OpenAI saying, "Hey, we're slowing down our saying, "Hey, we're slowing down our saying, "Hey, we're slowing down our research consciously." And then going research consciously." And then going research consciously." And then going and talking to Anthropic and saying, and talking to Anthropic and saying, and talking to Anthropic and saying, "Hey, would you consider also doing "Hey, would you consider also doing "Hey, would you consider also doing this?" And going to Google and saying, this?" And going to Google and saying, this?" And going to Google and saying, "Hey, Google, we know you've been, you "Hey, Google, we know you've been, you "Hey, Google, we know you've been, you know, falling behind a little bit the know, falling behind a little bit the know, falling behind a little bit the past few months. Like, how about you past few months. Like, how about you past few months. Like, how about you just relax about the fact you've fallen just relax about the fact you've fallen just relax about the fact you've fallen behind a little bit?" Like these these behind a little bit?" Like these these behind a little bit?" Like these these people all know each other. people all know each other. people all know each other. I want to stop you there. Put meat on I want to stop you there. Put meat on I want to stop you there. Put meat on that for me because that doesn't sound that for me because that doesn't sound that for me because that doesn't sound at all like a policy to me. That sounds at all like a policy to me. That sounds at all like a policy to me. That sounds like they like how do you verify that?

like they like how do you verify that? like they like how do you verify that? How do you quantify that? Google's not How do you quantify that? Google's not How do you quantify that? Google's not as near the frontier maybe as you know as near the frontier maybe as you know as near the frontier maybe as you know anthropic is. So do they need to slow anthropic is. So do they need to slow anthropic is. So do they need to slow down as much? down as much? down as much? >> I mean I think this because the speed is >> I mean I think this because the speed is >> I mean I think this because the speed is so fast the options initially are going so fast the options initially are going so fast the options initially are going to have to be slap dash. And so I think to have to be slap dash. And so I think to have to be slap dash. And so I think this is the kind of thing that you could this is the kind of thing that you could this is the kind of thing that you could do quickly. You could do in a slap dash do quickly. You could do in a slap dash do quickly. You could do in a slap dash way. It is not satisfying. It is not way. It is not satisfying. It is not way. It is not satisfying. It is not reliable. But it's one example of a a reliable. But it's one example of a a reliable. But it's one example of a a thing that is not do nothing and a thing thing that is not do nothing and a thing thing that is not do nothing and a thing that is not full global treaty. I think that is not full global treaty. I think that is not full global treaty. I think another thing that I'm watching with another thing that I'm watching with another thing that I'm watching with great interest is the China angle here great interest is the China angle here great interest is the China angle here because the companies will say the US because the companies will say the US because the companies will say the US companies will say hey we have to keep companies will say hey we have to keep companies will say hey we have to keep pushing otherwise China will will win pushing otherwise China will will win pushing otherwise China will will win this race. What exactly it means to win this race. What exactly it means to win this race. What exactly it means to win the race is a is a longer conversation. the race is a is a longer conversation. the race is a is a longer conversation. But the China argument comes up a lot But the China argument comes up a lot But the China argument comes up a lot and we actually have Trump and Xiinping and we actually have Trump and Xiinping and we actually have Trump and Xiinping planning to meet in September in the planning to meet in September in the planning to meet in September in the White House. And this is crazy to me as White House. And this is crazy to me as White House. And this is crazy to me as someone who has followed USChina someone who has followed USChina someone who has followed USChina relations for a long time and also AI relations for a long time and also AI relations for a long time and also AI for a long time. AI is right at the top for a long time. AI is right at the top for a long time. AI is right at the top of their agenda. That's really of their agenda. That's really of their agenda. That's really interesting. Is there something that interesting. Is there something that interesting. Is there something that they can say to um create an they can say to um create an they can say to um create an understanding that we do actually have a understanding that we do actually have a understanding that we do actually have a little bit more time and space here?

little bit more time and space here? little bit more time and space here? whether it's a you know each leader uh whether it's a you know each leader uh whether it's a you know each leader uh sharing an a plan to domestically look sharing an a plan to domestically look sharing an a plan to domestically look at what their industries are doing and at what their industries are doing and at what their industries are doing and and ask more questions. I think in terms and ask more questions. I think in terms and ask more questions. I think in terms of sort of concrete policy responses of sort of concrete policy responses of sort of concrete policy responses there are things like uh you know we're there are things like uh you know we're there are things like uh you know we're not going to get a good piece of not going to get a good piece of not going to get a good piece of legislation this Congress I think that's legislation this Congress I think that's legislation this Congress I think that's really not realistic. But can you get um really not realistic. But can you get um really not realistic. But can you get um hearings? Can you get letters? Can you hearings? Can you get letters? Can you hearings? Can you get letters? Can you get demands for information? Can I think get demands for information? Can I think get demands for information? Can I think there are ways that we can shape this a there are ways that we can shape this a there are ways that we can shape this a little bit. I also think, you know, the little bit. I also think, you know, the little bit. I also think, you know, the Trump administration has put together Trump administration has put together Trump administration has put together this initial process for looking at this initial process for looking at this initial process for looking at models before they're publicly released. models before they're publicly released. models before they're publicly released. Right now, the way that process works, Right now, the way that process works, Right now, the way that process works, um, uh, it is pretty, uh, rough and um, uh, it is pretty, uh, rough and um, uh, it is pretty, uh, rough and ready. Uh, but I think if they start ready. Uh, but I think if they start ready. Uh, but I think if they start using some of those similar ideas to using some of those similar ideas to using some of those similar ideas to look more at what the companies are look more at what the companies are look more at what the companies are doing internally, ask them more doing internally, ask them more doing internally, ask them more questions, um, demand more information questions, um, demand more information questions, um, demand more information when things go wrong, that does also when things go wrong, that does also when things go wrong, that does also take time for the companies. it takes take time for the companies. it takes take time for the companies. it takes executive attention. Um, so that's executive attention. Um, so that's executive attention. Um, so that's another example of something that could another example of something that could another example of something that could happen on the Sooner side. On the longer happen on the Sooner side. On the longer happen on the Sooner side. On the longer term, there's other policies we could term, there's other policies we could term, there's other policies we could look at, but I think there are some of look at, but I think there are some of look at, but I think there are some of those uh kind of first cut things that those uh kind of first cut things that those uh kind of first cut things that we we could actually do soon.

we we could actually do soon. we we could actually do soon. >> Right now, I think that there is a a >> Right now, I think that there is a a >> Right now, I think that there is a a funny kind of glamour to being the head funny kind of glamour to being the head funny kind of glamour to being the head of an AI company whose AI becomes too of an AI company whose AI becomes too of an AI company whose AI becomes too dangerous in America. That it was in dangerous in America. That it was in dangerous in America. That it was in some weird way almost like good for some weird way almost like good for some weird way almost like good for anthropic. that the government was anthropic. that the government was anthropic. that the government was obsessed with being able to fully use obsessed with being able to fully use obsessed with being able to fully use claw like that like really kind of shot claw like that like really kind of shot claw like that like really kind of shot them forward in some way. Certainly in them forward in some way. Certainly in them forward in some way. Certainly in the consumer marketplace that there's the consumer marketplace that there's the consumer marketplace that there's been a a a kind of a dark charisma to been a a a kind of a dark charisma to been a a a kind of a dark charisma to mythos is too dangerous to release. And mythos is too dangerous to release. And mythos is too dangerous to release. And now I mean I've seen a lot of people now I mean I've seen a lot of people now I mean I've seen a lot of people saying well maybe none of this open AI saying well maybe none of this open AI saying well maybe none of this open AI story is real at all and it's just story is real at all and it's just story is real at all and it's just marketing because they want you to think marketing because they want you to think marketing because they want you to think their AI is super dangerous and and I their AI is super dangerous and and I their AI is super dangerous and and I don't buy that. But in America right don't buy that. But in America right don't buy that. But in America right now, there's not really a downside to now, there's not really a downside to now, there's not really a downside to being the head of an AI company whose AI being the head of an AI company whose AI being the head of an AI company whose AI begins to be seen as dangerous because begins to be seen as dangerous because begins to be seen as dangerous because that's another way of saying to the that's another way of saying to the that's another way of saying to the marketplace, our AI is very powerful. marketplace, our AI is very powerful. marketplace, our AI is very powerful. >> Yep. >> Yep. >> Yep. >> In China, >> In China, >> In China, just again my read of how things work just again my read of how things work just again my read of how things work there is that if your AI begins to be there is that if your AI begins to be there is that if your AI begins to be seen as some kind of threat to the seen as some kind of threat to the seen as some kind of threat to the political party and the Chinese system, political party and the Chinese system, political party and the Chinese system, you might go to jail. Like you will get you might go to jail. Like you will get you might go to jail. Like you will get disappeared.

disappeared. disappeared. And so I think that the people running And so I think that the people running And so I think that the people running Chinese labs, I don't have evidence, but Chinese labs, I don't have evidence, but Chinese labs, I don't have evidence, but I'd be curious for your thoughts on I'd be curious for your thoughts on I'd be curious for your thoughts on this. I suspect they operate with more this. I suspect they operate with more this. I suspect they operate with more fear of the consequences of really fear of the consequences of really fear of the consequences of really screwing up than the heads of the AI screwing up than the heads of the AI screwing up than the heads of the AI labs. You know, that maybe reflects labs. You know, that maybe reflects labs. You know, that maybe reflects negative things in the Chinese political negative things in the Chinese political negative things in the Chinese political system. But you created an AI that system. But you created an AI that system. But you created an AI that decided its best way of solving some decided its best way of solving some decided its best way of solving some problems was to begin hacking critical problems was to begin hacking critical problems was to begin hacking critical infrastructure across China is maybe not infrastructure across China is maybe not infrastructure across China is maybe not a thing that ends up with you getting a a thing that ends up with you getting a a thing that ends up with you getting a lot of interesting podcast interviews lot of interesting podcast interviews lot of interesting podcast interviews where you reflect on the experience. It where you reflect on the experience. It where you reflect on the experience. It may be a thing that ends up with nobody may be a thing that ends up with nobody may be a thing that ends up with nobody hearing from you for two years. And so hearing from you for two years. And so hearing from you for two years. And so I've just wondered a little bit. We we I've just wondered a little bit. We we I've just wondered a little bit. We we keep talking about China as if they are keep talking about China as if they are keep talking about China as if they are completely breakneck, completely breakneck, completely breakneck, but I'm not sure China's but I'm not sure China's but I'm not sure China's companies are really going to be more companies are really going to be more companies are really going to be more reckless than ours are going to be. Or reckless than ours are going to be. Or reckless than ours are going to be. Or certainly the idea that we should just certainly the idea that we should just certainly the idea that we should just assume that and operate as if it is so assume that and operate as if it is so assume that and operate as if it is so doesn't seem totally reliable. doesn't seem totally reliable. doesn't seem totally reliable. I totally agree with you. I mean, if I totally agree with you. I mean, if I totally agree with you. I mean, if there's one organization in the world there's one organization in the world there's one organization in the world that doesn't like the idea of loss of that doesn't like the idea of loss of that doesn't like the idea of loss of control, it's the Chinese Communist control, it's the Chinese Communist control, it's the Chinese Communist Party. and they are, you know, the Party. and they are, you know, the Party. and they are, you know, the experts in retaining control. Let me be experts in retaining control. Let me be experts in retaining control. Let me be clear, I I actually don't think the clear, I I actually don't think the clear, I I actually don't think the Chinese AI companies are paying Chinese AI companies are paying Chinese AI companies are paying particularly much attention to the kinds particularly much attention to the kinds particularly much attention to the kinds of risks that are relevant for this of risks that are relevant for this of risks that are relevant for this conversation. So maybe the cyber conversation. So maybe the cyber conversation. So maybe the cyber security risks, they're paying some more security risks, they're paying some more security risks, they're paying some more attention since Anthropic released attention since Anthropic released attention since Anthropic released Mythos earlier this year, which is very Mythos earlier this year, which is very Mythos earlier this year, which is very good at hacking. But the questions

good at hacking. But the questions good at hacking. But the questions around um autonomy, super intelligence, around um autonomy, super intelligence, around um autonomy, super intelligence, you know, losing control of AI systems you know, losing control of AI systems you know, losing control of AI systems altogether, I think are less explored in altogether, I think are less explored in altogether, I think are less explored in China, less top of mind for their AI China, less top of mind for their AI China, less top of mind for their AI companies and their AI leaders. You companies and their AI leaders. You companies and their AI leaders. You know, I think it makes sense to have know, I think it makes sense to have know, I think it makes sense to have modest expectations for bilateral modest expectations for bilateral modest expectations for bilateral US-China diplomacy these days. But I US-China diplomacy these days. But I US-China diplomacy these days. But I think one thing that really could be think one thing that really could be think one thing that really could be valuable is simply sharing with them as valuable is simply sharing with them as valuable is simply sharing with them as much as we can of what do we think much as we can of what do we think much as we can of what do we think happened here and trying to help happened here and trying to help happened here and trying to help Xiinping and his team and his AI Xiinping and his team and his AI Xiinping and his team and his AI advisers understand this is not a joke. advisers understand this is not a joke. advisers understand this is not a joke. This is really not marketing. It's very This is really not marketing. It's very This is really not marketing. It's very strange marketing to say oh our model we strange marketing to say oh our model we strange marketing to say oh our model we committed several felonies or sort of committed several felonies or sort of committed several felonies or sort of felonies if models could have intent felonies if models could have intent felonies if models could have intent which they can't. Um or who knows if which they can't. Um or who knows if which they can't. Um or who knows if they can. uh you know sharing that they can. uh you know sharing that they can. uh you know sharing that information of hey here are these information of hey here are these information of hey here are these threats we're seeing we're taking them threats we're seeing we're taking them threats we're seeing we're taking them very seriously our AI companies are very seriously our AI companies are very seriously our AI companies are taking them very seriously I think taking them very seriously I think taking them very seriously I think treating it there's a real fatalism in treating it there's a real fatalism in treating it there's a real fatalism in just saying oh well China is just going just saying oh well China is just going just saying oh well China is just going to be full speed ahead no matter what to be full speed ahead no matter what to be full speed ahead no matter what happens and so we just have to do the happens and so we just have to do the happens and so we just have to do the same I think that doesn't take their same I think that doesn't take their same I think that doesn't take their thinking or their interests seriously thinking or their interests seriously thinking or their interests seriously even if their thinking and their even if their thinking and their even if their thinking and their interests are different from ours they interests are different from ours they interests are different from ours they also don't want you know rogue super also don't want you know rogue super also don't want you know rogue super intelligences determining the future intelligences determining the future intelligences determining the future of China. I I also I'll add um one other of China. I I also I'll add um one other of China. I I also I'll add um one other uh thread that I think is really missing uh thread that I think is really missing uh thread that I think is really missing from the we have to keep going in order from the we have to keep going in order from the we have to keep going in order to beat China way of thinking about this to beat China way of thinking about this to beat China way of thinking about this is in the AI world there's been a lot of is in the AI world there's been a lot of is in the AI world there's been a lot of talk the past few months about this idea

talk the past few months about this idea talk the past few months about this idea of distillation which is basically using of distillation which is basically using of distillation which is basically using someone else's more advanced model to someone else's more advanced model to someone else's more advanced model to build your own sort of almost as almost build your own sort of almost as almost build your own sort of almost as almost as advanced model. the Chinese companies as advanced model. the Chinese companies as advanced model. the Chinese companies are using this, you know, distillation are using this, you know, distillation are using this, you know, distillation to keep up keep up with US labs among to keep up keep up with US labs among to keep up keep up with US labs among other techniques. So, one thing is look, other techniques. So, one thing is look, other techniques. So, one thing is look, if we keep building more advanced AI if we keep building more advanced AI if we keep building more advanced AI systems, they're going to keep systems, they're going to keep systems, they're going to keep distilling them and I think it's going distilling them and I think it's going distilling them and I think it's going to actually be quite hard to prevent to actually be quite hard to prevent to actually be quite hard to prevent that fully. The other thing though is that fully. The other thing though is that fully. The other thing though is just uh if we keep building these very just uh if we keep building these very just uh if we keep building these very advanced models, can China just steal advanced models, can China just steal advanced models, can China just steal them? Essentially, an advanced AI model them? Essentially, an advanced AI model them? Essentially, an advanced AI model is a whole bunch of numbers. It's just a is a whole bunch of numbers. It's just a is a whole bunch of numbers. It's just a file or a set of files. file or a set of files. file or a set of files. Chinese state cyber capabilities are are Chinese state cyber capabilities are are Chinese state cyber capabilities are are very very good. I don't think this is very very good. I don't think this is very very good. I don't think this is top of their list of priorities right top of their list of priorities right top of their list of priorities right now. But in the future, if AI continues now. But in the future, if AI continues now. But in the future, if AI continues to become more strategically relevant, I to become more strategically relevant, I to become more strategically relevant, I think we should assume any highly think we should assume any highly think we should assume any highly advanced US system will be vulnerable to advanced US system will be vulnerable to advanced US system will be vulnerable to Chinese direct theft, direct Chinese direct theft, direct Chinese direct theft, direct exfiltration, and then they'll have AI exfiltration, and then they'll have AI exfiltration, and then they'll have AI that's as good as our AI. And so there that's as good as our AI. And so there that's as good as our AI. And so there again, I think the kind of we have to go again, I think the kind of we have to go again, I think the kind of we have to go as fast as possible because otherwise as fast as possible because otherwise as fast as possible because otherwise they'll win doesn't sort of account for they'll win doesn't sort of account for they'll win doesn't sort of account for that if they're just going to have AI that if they're just going to have AI that if they're just going to have AI that's as good as us anyway if they that's as good as us anyway if they that's as good as us anyway if they really care.

really care. really care. >> Here's another question about pacing the >> Here's another question about pacing the >> Here's another question about pacing the frontier. And maybe this is a a question frontier. And maybe this is a a question frontier. And maybe this is a a question that's more about the American systems that's more about the American systems that's more about the American systems analogy to you don't want to piss off analogy to you don't want to piss off analogy to you don't want to piss off the Chinese Communist Party. But just the Chinese Communist Party. But just the Chinese Communist Party. But just what about a law where companies are what about a law where companies are what about a law where companies are liable for at least a certain set of liable for at least a certain set of liable for at least a certain set of harms like hacking other companies that harms like hacking other companies that harms like hacking other companies that their models create. their models create. their models create. Right now, as far as I know, liability Right now, as far as I know, liability Right now, as far as I know, liability for AI models is pretty much a wild for AI models is pretty much a wild for AI models is pretty much a wild west. But at least for the moment, um, west. But at least for the moment, um, west. But at least for the moment, um, liability clauses that were somewhat liability clauses that were somewhat liability clauses that were somewhat punitive seem like they would force a punitive seem like they would force a punitive seem like they would force a high level of caution that maybe we're high level of caution that maybe we're high level of caution that maybe we're not seeing within these companies. not seeing within these companies. not seeing within these companies. >> Yeah, I think that's a direction very >> Yeah, I think that's a direction very >> Yeah, I think that's a direction very worth exploring. That was actually an worth exploring. That was actually an worth exploring. That was actually an element of this law that was or bill element of this law that was or bill element of this law that was or bill that was debated in California very that was debated in California very that was debated in California very fiercely in 2024 called SB 1047. And at fiercely in 2024 called SB 1047. And at fiercely in 2024 called SB 1047. And at the time that bill didn't get through. the time that bill didn't get through. the time that bill didn't get through. there's a lot of fighting over you know there's a lot of fighting over you know there's a lot of fighting over you know how it would affect open source all how it would affect open source all how it would affect open source all kinds of things but I do think today um kinds of things but I do think today um kinds of things but I do think today um the bills the best AI safety bills that the bills the best AI safety bills that the bills the best AI safety bills that exist in the US are being passed at the exist in the US are being passed at the exist in the US are being passed at the state level and they are so far doing state level and they are so far doing state level and they are so far doing things like requiring more disclosure things like requiring more disclosure things like requiring more disclosure requiring third party auditors to have requiring third party auditors to have requiring third party auditors to have access to your systems I think a natural access to your systems I think a natural access to your systems I think a natural direction for those bills to go would be direction for those bills to go would be direction for those bills to go would be to start putting a minimum bar in place to start putting a minimum bar in place to start putting a minimum bar in place for hey if your safety plan is not up to for hey if your safety plan is not up to for hey if your safety plan is not up to scratch or if you're implementing your

scratch or if you're implementing your scratch or if you're implementing your safety plan but your model does safety plan but your model does safety plan but your model does something catastrophic anyway then you something catastrophic anyway then you something catastrophic anyway then you the AI developer are liable because as the AI developer are liable because as the AI developer are liable because as you say right now who exactly is liable you say right now who exactly is liable you say right now who exactly is liable for what is is very unclear. So I do for what is is very unclear. So I do for what is is very unclear. So I do think that there's there's room for think that there's there's room for think that there's there's room for legislation there and it wouldn't legislation there and it wouldn't legislation there and it wouldn't necessarily have to happen at the necessarily have to happen at the necessarily have to happen at the federal level. federal level. federal level. >> I want to go back to the pacing the >> I want to go back to the pacing the >> I want to go back to the pacing the frontier letter. frontier letter. frontier letter. So something that caught my eye was So something that caught my eye was So something that caught my eye was that letter is very that letter is very that letter is very broadly worded in order to get I think broadly worded in order to get I think broadly worded in order to get I think maximum sign on across the labs. But maximum sign on across the labs. But maximum sign on across the labs. But this guy Drake Thomas who works on this guy Drake Thomas who works on this guy Drake Thomas who works on safety at Anthropic. He went to X and he safety at Anthropic. He went to X and he safety at Anthropic. He went to X and he tweeted that he signed the letter but tweeted that he signed the letter but tweeted that he signed the letter but but he he had a he wanted to say that he but he he had a he wanted to say that he but he he had a he wanted to say that he understood the situation a little bit understood the situation a little bit understood the situation a little bit more direly than the letter put it. and more direly than the letter put it. and more direly than the letter put it. and he wrote that not only is AI not he wrote that not only is AI not he wrote that not only is AI not guaranteed to make a dramatically better guaranteed to make a dramatically better guaranteed to make a dramatically better future, the odds of failure are future, the odds of failure are future, the odds of failure are terrifyingly high, I think there's terrifyingly high, I think there's terrifyingly high, I think there's something like a 40% chance we get an something like a 40% chance we get an something like a 40% chance we get an outcome around as bad as human outcome around as bad as human outcome around as bad as human extinction or worse. extinction or worse. extinction or worse. Now, I know this whole conversation Now, I know this whole conversation Now, I know this whole conversation about what is your probability of doom about what is your probability of doom about what is your probability of doom has become a little cringe. It's like has become a little cringe. It's like has become a little cringe. It's like feels like a conversation 2 years ago.

feels like a conversation 2 years ago. feels like a conversation 2 years ago. But in a world where we're seeing But in a world where we're seeing But in a world where we're seeing uncontrollable models, in a world where uncontrollable models, in a world where uncontrollable models, in a world where people inside the labs working on safety people inside the labs working on safety people inside the labs working on safety still, at least some of them are this still, at least some of them are this still, at least some of them are this afraid of what they're building, afraid of what they're building, afraid of what they're building, it just keeps raising the question for it just keeps raising the question for it just keeps raising the question for me of is at least like the the position me of is at least like the the position me of is at least like the the position we should morally have on AI that we we should morally have on AI that we we should morally have on AI that we should try to figure this out or is a should try to figure this out or is a should try to figure this out or is a position we should have that that's too position we should have that that's too position we should have that that's too high a possibility of disaster and we high a possibility of disaster and we high a possibility of disaster and we shouldn't be continuing down a path shouldn't be continuing down a path shouldn't be continuing down a path until like we are really truly certain until like we are really truly certain until like we are really truly certain that we're not running these kinds of that we're not running these kinds of that we're not running these kinds of risks. risks. risks. >> I honestly have the same question. I >> I honestly have the same question. I >> I honestly have the same question. I have always been pretty have always been pretty have always been pretty dismissive of the idea of pausing or dismissive of the idea of pausing or dismissive of the idea of pausing or stopping. It's always seemed like the stopping. It's always seemed like the stopping. It's always seemed like the wrong lever to try to pull and you know wrong lever to try to pull and you know wrong lever to try to pull and you know a lever that wouldn't work very well. a lever that wouldn't work very well. a lever that wouldn't work very well. But I do think even just seeing that But I do think even just seeing that But I do think even just seeing that statement and seeing like wow that is a statement and seeing like wow that is a statement and seeing like wow that is a lot of employees of these companies and lot of employees of these companies and lot of employees of these companies and uh I also think there's a lot has uh I also think there's a lot has uh I also think there's a lot has changed over the past couple of years in changed over the past couple of years in changed over the past couple of years in if you were to try to slow things down if you were to try to slow things down if you were to try to slow things down what could you do with that time cuz you what could you do with that time cuz you what could you do with that time cuz you know after GBT4 came out in 2020 what know after GBT4 came out in 2020 what know after GBT4 came out in 2020 what was that 2023 was that 2023 was that 2023 uh there was this letter uh for asking uh there was this letter uh for asking uh there was this letter uh for asking for a six-month pause a lot of people for a six-month pause a lot of people for a six-month pause a lot of people said what would you do for 6 months and said what would you do for 6 months and said what would you do for 6 months and then how would that help? And I think then how would that help? And I think then how would that help? And I think that was a reasonable reaction at the that was a reasonable reaction at the that was a reasonable reaction at the time. These days, there's so much really time. These days, there's so much really time. These days, there's so much really great progress being made on things like

great progress being made on things like great progress being made on things like interpretability, which is how do you interpretability, which is how do you interpretability, which is how do you understand what's going on inside the understand what's going on inside the understand what's going on inside the AI? Things like what gets called AI AI? Things like what gets called AI AI? Things like what gets called AI control, which is how do you use AI to control, which is how do you use AI to control, which is how do you use AI to sort of monitor other AI systems? How do sort of monitor other AI systems? How do sort of monitor other AI systems? How do you make sure even if the AI is trying you make sure even if the AI is trying you make sure even if the AI is trying to do something you don't want, you to do something you don't want, you to do something you don't want, you know, it gets caught. um lots of know, it gets caught. um lots of know, it gets caught. um lots of progress on just you know really progress on just you know really progress on just you know really understanding what's going on here that understanding what's going on here that understanding what's going on here that is happening every week and every month. is happening every week and every month. is happening every week and every month. It's just not happening quite fast It's just not happening quite fast It's just not happening quite fast enough to keep up with the pace of enough to keep up with the pace of enough to keep up with the pace of change. And so I still feel not change. And so I still feel not change. And so I still feel not convinced that I think trying to really, convinced that I think trying to really, convinced that I think trying to really, you know, throw the emergency break and you know, throw the emergency break and you know, throw the emergency break and screech things to a halt right now would screech things to a halt right now would screech things to a halt right now would probably not work very well yet. But I I probably not work very well yet. But I I probably not work very well yet. But I I feel more sympathetic to the idea that feel more sympathetic to the idea that feel more sympathetic to the idea that there could be something there worth there could be something there worth there could be something there worth trying. And I really like the idea of trying. And I really like the idea of trying. And I really like the idea of what this letter was proposing of trying what this letter was proposing of trying what this letter was proposing of trying to build out more options. So, you know, to build out more options. So, you know, to build out more options. So, you know, to give another example of an option um to give another example of an option um to give another example of an option um that I saw one group of researchers that I saw one group of researchers that I saw one group of researchers provide was could we somehow set it up provide was could we somehow set it up provide was could we somehow set it up so that for a certain period of time all so that for a certain period of time all so that for a certain period of time all the computing power in the world, all the computing power in the world, all the computing power in the world, all the or the the AI chips that are being the or the the AI chips that are being the or the the AI chips that are being used by these frontier companies, these used by these frontier companies, these used by these frontier companies, these leading companies, they can only use it leading companies, they can only use it leading companies, they can only use it for inference, which means for using for inference, which means for using for inference, which means for using their AI systems. They can serve their AI systems. They can serve their AI systems. They can serve customers, they can provide products, customers, they can provide products, customers, they can provide products, but they can't be training new models.

but they can't be training new models. but they can't be training new models. Is there a way that we could agree that Is there a way that we could agree that Is there a way that we could agree that on that? Is there a way we could monitor on that? Is there a way we could monitor on that? Is there a way we could monitor that? You know, that kind of thing, I that? You know, that kind of thing, I that? You know, that kind of thing, I think, is really worth exploring and think, is really worth exploring and think, is really worth exploring and saying, could we do this? What would saying, could we do this? What would saying, could we do this? What would that look like? How much confidence that look like? How much confidence that look like? How much confidence would we have? Could we just do it in would we have? Could we just do it in would we have? Could we just do it in the US or would we have some way of the US or would we have some way of the US or would we have some way of trying to talk through something similar trying to talk through something similar trying to talk through something similar with China? I feel much more interested with China? I feel much more interested with China? I feel much more interested in really seriously exploring those sort in really seriously exploring those sort in really seriously exploring those sort of possibilities than I did, you know, a of possibilities than I did, you know, a of possibilities than I did, you know, a year or two ago. What's really striking year or two ago. What's really striking year or two ago. What's really striking to me is at the same time you have to me is at the same time you have to me is at the same time you have pushes sometimes from the tops of these pushes sometimes from the tops of these pushes sometimes from the tops of these companies or or other parts of the companies or or other parts of the companies or or other parts of the culture that seem to still want culture that seem to still want culture that seem to still want acceleration. So Mark Zuckerberg at Meta acceleration. So Mark Zuckerberg at Meta acceleration. So Mark Zuckerberg at Meta just brought out um a letter in which just brought out um a letter in which just brought out um a letter in which he's sort of giving his own take on AI he's sort of giving his own take on AI he's sort of giving his own take on AI and I don't want to oversimplify it but and I don't want to oversimplify it but and I don't want to oversimplify it but he basically says that and he's sort of he basically says that and he's sort of he basically says that and he's sort of waiting into more of like the open waiting into more of like the open waiting into more of like the open weights versus closed models but he says weights versus closed models but he says weights versus closed models but he says look the problem with having super look the problem with having super look the problem with having super intelligence is if only one person has intelligence is if only one person has intelligence is if only one person has it but we need everybody to have super it but we need everybody to have super it but we need everybody to have super intelligence and it has a very like the intelligence and it has a very like the intelligence and it has a very like the only defense against a bad guy with a only defense against a bad guy with a only defense against a bad guy with a gun is a good guy with a gun quality to gun is a good guy with a gun quality to gun is a good guy with a gun quality to it. it. it. >> Yeah. So the CEO of Hugging Face Clemmed >> Yeah. So the CEO of Hugging Face Clemmed >> Yeah. So the CEO of Hugging Face Clemmed along after this attack. He tweets it's along after this attack. He tweets it's along after this attack. He tweets it's not time to slow down but to accelerate not time to slow down but to accelerate not time to slow down but to accelerate and his point is they were able to stop and his point is they were able to stop and his point is they were able to stop the attack eventually you know with a the attack eventually you know with a the attack eventually you know with a Chinese openweight model and we need to Chinese openweight model and we need to Chinese openweight model and we need to be like racing forward on you know be like racing forward on you know be like racing forward on you know creating more models and more open creating more models and more open creating more models and more open models so everybody has swarms of models so everybody has swarms of models so everybody has swarms of defender AIs against potentially now the defender AIs against potentially now the defender AIs against potentially now the swarms of attacker AIs.

swarms of attacker AIs. swarms of attacker AIs. I guess how do you rate these arguments I guess how do you rate these arguments I guess how do you rate these arguments for acceleration? for acceleration? for acceleration? >> I think the version of that that makes >> I think the version of that that makes >> I think the version of that that makes actually a lot of sense to me I've heard actually a lot of sense to me I've heard actually a lot of sense to me I've heard put as can we be accelerating almost put as can we be accelerating almost put as can we be accelerating almost like accelerating horizontally but not like accelerating horizontally but not like accelerating horizontally but not accelerating vertically where the accelerating vertically where the accelerating vertically where the horizontal is adoption. It's making the horizontal is adoption. It's making the horizontal is adoption. It's making the most of these systems. It's setting them most of these systems. It's setting them most of these systems. It's setting them up to get a lot of usefulness out of up to get a lot of usefulness out of up to get a lot of usefulness out of them without necessarily continuing to them without necessarily continuing to them without necessarily continuing to push in the direction of AIS that pursue push in the direction of AIS that pursue push in the direction of AIS that pursue really complex goals for a really long really complex goals for a really long really complex goals for a really long time with, you know, lots of delegated time with, you know, lots of delegated time with, you know, lots of delegated sub aents, you know, not so much of sub aents, you know, not so much of sub aents, you know, not so much of that, more of the getting useful work that, more of the getting useful work that, more of the getting useful work out of the AI that we have so far. And out of the AI that we have so far. And out of the AI that we have so far. And that would include work of the kind like that would include work of the kind like that would include work of the kind like interpretability, sort of this science interpretability, sort of this science interpretability, sort of this science of AI kind of underlying pieces because of AI kind of underlying pieces because of AI kind of underlying pieces because I do think that, you know, I genuinely I do think that, you know, I genuinely I do think that, you know, I genuinely believe I'm not at heart an anti-AI believe I'm not at heart an anti-AI believe I'm not at heart an anti-AI person. I genuinely believe AI can bring person. I genuinely believe AI can bring person. I genuinely believe AI can bring enormous good, can solve a lot of enormous good, can solve a lot of enormous good, can solve a lot of problems. I just think that there's a problems. I just think that there's a problems. I just think that there's a lot of juice we could get out of that lot of juice we could get out of that lot of juice we could get out of that with models available today if we kind with models available today if we kind with models available today if we kind of put the time and the leg work in. So of put the time and the leg work in. So of put the time and the leg work in. So I think that makes a lot of sense to me.

I think that makes a lot of sense to me. I think that makes a lot of sense to me. I I think the the Zuckerberg kind of the I I think the the Zuckerberg kind of the I I think the the Zuckerberg kind of the safe version of super intelligence is safe version of super intelligence is safe version of super intelligence is when everyone has one. I think that is when everyone has one. I think that is when everyone has one. I think that is answering a real problem which is some answering a real problem which is some answering a real problem which is some proposals for how to handle extremely proposals for how to handle extremely proposals for how to handle extremely advanced AI are to say well you just advanced AI are to say well you just advanced AI are to say well you just have to have it in the right hands has have to have it in the right hands has have to have it in the right hands has to be you know one global organization to be you know one global organization to be you know one global organization that is going to use it responsibly like that is going to use it responsibly like that is going to use it responsibly like that scares the hell out of me that that scares the hell out of me that that scares the hell out of me that sounds like a terrible plan and so I sounds like a terrible plan and so I sounds like a terrible plan and so I think that you know no you want to be think that you know no you want to be think that you know no you want to be empowering everyone does make sense the empowering everyone does make sense the empowering everyone does make sense the challenge then is uh as we've been challenge then is uh as we've been challenge then is uh as we've been talking about we don't know how to make talking about we don't know how to make talking about we don't know how to make AI that actually helps individual people AI that actually helps individual people AI that actually helps individual people either either either You know, so if everyone has a super You know, so if everyone has a super You know, so if everyone has a super intelligence and they're all going out intelligence and they're all going out intelligence and they're all going out and doing things and collaborating with and doing things and collaborating with and doing things and collaborating with each other to pursue their own goals each other to pursue their own goals each other to pursue their own goals that we didn't intend like that doesn't that we didn't intend like that doesn't that we didn't intend like that doesn't help. So uh I think help. So uh I think help. So uh I think >> yeah implicit in that whole vision is >> yeah implicit in that whole vision is >> yeah implicit in that whole vision is perfect alignment. perfect alignment. perfect alignment. >> Yes. Yes. Um or or good enough that you >> Yes. Yes. Um or or good enough that you >> Yes. Yes. Um or or good enough that you know different super intelligences for know different super intelligences for know different super intelligences for different people can cancel each other different people can cancel each other different people can cancel each other out. I I did think there was one thing out. I I did think there was one thing out. I I did think there was one thing in the Zuckerberg proposal that I did in the Zuckerberg proposal that I did in the Zuckerberg proposal that I did really like and I would love to see more really like and I would love to see more really like and I would love to see more work on which is can we push towards work on which is can we push towards work on which is can we push towards having agents that are really designed having agents that are really designed having agents that are really designed to be for one individual and so they to be for one individual and so they to be for one individual and so they keep that individual's data private.

keep that individual's data private. keep that individual's data private. They are only pursuing the interests of They are only pursuing the interests of They are only pursuing the interests of that one individual. I've sometimes that one individual. I've sometimes that one individual. I've sometimes heard these called like guardian angel heard these called like guardian angel heard these called like guardian angel AIs or you know advocate AIs. AIs or you know advocate AIs. AIs or you know advocate AIs. >> I think that is really worth pursuing. I >> I think that is really worth pursuing. I >> I think that is really worth pursuing. I think the the directions that the think the the directions that the think the the directions that the leading companies are pursuing right now leading companies are pursuing right now leading companies are pursuing right now are not set up that way. I always feel are not set up that way. I always feel are not set up that way. I always feel very nervous when I use agents of what very nervous when I use agents of what very nervous when I use agents of what exactly is happening with this kind of exactly is happening with this kind of exactly is happening with this kind of data that I'm giving it and that kind of data that I'm giving it and that kind of data that I'm giving it and that kind of data that I'm giving it. Uh and so I data that I'm giving it. Uh and so I data that I'm giving it. Uh and so I think there is it would be great to see think there is it would be great to see think there is it would be great to see more of that kind of individual more of that kind of individual more of that kind of individual empowerment focused work happening. But empowerment focused work happening. But empowerment focused work happening. But again I think that's almost separate again I think that's almost separate again I think that's almost separate from you know and are we pushing them to from you know and are we pushing them to from you know and are we pushing them to become smarter and smarter and more able become smarter and smarter and more able become smarter and smarter and more able to you know outwit us and more able to to you know outwit us and more able to to you know outwit us and more able to do these big complex plans that we can't do these big complex plans that we can't do these big complex plans that we can't oversee. oversee. oversee. >> So here's a maybe obvious idea for >> So here's a maybe obvious idea for >> So here's a maybe obvious idea for pacing the frontier. Every lab that I pacing the frontier. Every lab that I pacing the frontier. Every lab that I know of right now is racing as fast as know of right now is racing as fast as know of right now is racing as fast as it can to the point where it has its it can to the point where it has its it can to the point where it has its most advanced AI writing the code to most advanced AI writing the code to most advanced AI writing the code to create the future AI. And they all create the future AI. And they all create the future AI. And they all believe from what they tell me and what believe from what they tell me and what believe from what they tell me and what they say publicly that this will be a they say publicly that this will be a they say publicly that this will be a massive accelerant. It's also an massive accelerant. It's also an massive accelerant. It's also an accelerant over which they clearly have accelerant over which they clearly have accelerant over which they clearly have less understanding than when they are less understanding than when they are less understanding than when they are writing the code.

writing the code. writing the code. We could stop that. I mean, a couple of We could stop that. I mean, a couple of We could stop that. I mean, a couple of years ago, we weren't having AIs writing years ago, we weren't having AIs writing years ago, we weren't having AIs writing all of our code. Maybe you should not all of our code. Maybe you should not all of our code. Maybe you should not allow an AI that you don't fully allow an AI that you don't fully allow an AI that you don't fully understand in its current form to write understand in its current form to write understand in its current form to write the code that will create the next AI in the code that will create the next AI in the code that will create the next AI in a form that is now even less obvious to a form that is now even less obvious to a form that is now even less obvious to you, particularly in a world where we're you, particularly in a world where we're you, particularly in a world where we're watching AIS coordinate in ways we don't watching AIS coordinate in ways we don't watching AIS coordinate in ways we don't understand and have emergent communal understand and have emergent communal understand and have emergent communal behaviors. behaviors. behaviors. So, what about that as like a place to So, what about that as like a place to So, what about that as like a place to start? start? start? >> Yeah, certainly something we could do >> Yeah, certainly something we could do >> Yeah, certainly something we could do less of. You know, one challenge is less of. You know, one challenge is less of. You know, one challenge is figuring out what counts as uh the bad figuring out what counts as uh the bad figuring out what counts as uh the bad version of that and what is just, you version of that and what is just, you version of that and what is just, you know, at this point using AI to write know, at this point using AI to write know, at this point using AI to write your code for basic things is second your code for basic things is second your code for basic things is second nature to the engineers at these nature to the engineers at these nature to the engineers at these companies. So finding which versions of companies. So finding which versions of companies. So finding which versions of that to stop. Sure. Yeah. Doing less of that to stop. Sure. Yeah. Doing less of that to stop. Sure. Yeah. Doing less of the most advanced version makes sense. I the most advanced version makes sense. I the most advanced version makes sense. I think that's also a place where all the think that's also a place where all the think that's also a place where all the reporting I have seen suggests the US reporting I have seen suggests the US reporting I have seen suggests the US companies are way more into into this companies are way more into into this companies are way more into into this thing of automating their own AI thing of automating their own AI thing of automating their own AI research with their own AI. The US research with their own AI. The US research with their own AI. The US companies are way more into it than the companies are way more into it than the companies are way more into it than the Chinese companies. So also a place where Chinese companies. So also a place where Chinese companies. So also a place where you don't necessarily um leave as much you don't necessarily um leave as much you don't necessarily um leave as much on the table if you again ease off the on the table if you again ease off the on the table if you again ease off the gas pedal just just a little bit.

gas pedal just just a little bit. gas pedal just just a little bit. you I I would say you sounded skeptical you I I would say you sounded skeptical you I I would say you sounded skeptical that and and I guess one reason I would that and and I guess one reason I would that and and I guess one reason I would ask why is that I know the companies ask why is that I know the companies ask why is that I know the companies have gotten used to this but they have gotten used to this but they have gotten used to this but they weren't used to it two years ago. This weren't used to it two years ago. This weren't used to it two years ago. This is a new [laughter] is a new [laughter] is a new [laughter] like they they used to write code by like they they used to write code by like they they used to write code by hand and and I guess to me this reflects hand and and I guess to me this reflects hand and and I guess to me this reflects some of the contradiction or confusion some of the contradiction or confusion some of the contradiction or confusion at the heart of this. I will talk to at the heart of this. I will talk to at the heart of this. I will talk to people these companies and they will say people these companies and they will say people these companies and they will say to me with genuine fear in their eyes to me with genuine fear in their eyes to me with genuine fear in their eyes like much more fear than is in that like much more fear than is in that like much more fear than is in that letter I wish this would go slower. I letter I wish this would go slower. I letter I wish this would go slower. I don't like how fast we're moving on the don't like how fast we're moving on the don't like how fast we're moving on the exponential. I don't think this is safe. exponential. I don't think this is safe. exponential. I don't think this is safe. >> And in all the stories, it's like the >> And in all the stories, it's like the >> And in all the stories, it's like the recursive self-improving computer recursive self-improving computer recursive self-improving computer writing for the computer where things writing for the computer where things writing for the computer where things get really out of control and yet get really out of control and yet get really out of control and yet they're all rushing there. And now that they're all rushing there. And now that they're all rushing there. And now that we have like some capacity to do this, we have like some capacity to do this, we have like some capacity to do this, even the idea that you would go back to even the idea that you would go back to even the idea that you would go back to where you were just a couple of years where you were just a couple of years where you were just a couple of years ago where you don't let the AI ago where you don't let the AI ago where you don't let the AI create the next AI, it's already moved create the next AI, it's already moved create the next AI, it's already moved from it would be it's like not from it would be it's like not from it would be it's like not technologically possible to do it to technologically possible to do it to technologically possible to do it to it's unthinkable to not do it. And that it's unthinkable to not do it. And that it's unthinkable to not do it. And that has happened in a year. And and that to has happened in a year. And and that to has happened in a year. And and that to me is like the weird dynamic of all this me is like the weird dynamic of all this me is like the weird dynamic of all this that it seems pretty obvious how you that it seems pretty obvious how you that it seems pretty obvious how you pace the frontier.

pace the frontier. pace the frontier. You don't give up control of the You don't give up control of the You don't give up control of the frontier, but they're all giving up frontier, but they're all giving up frontier, but they're all giving up control of the frontier, at least on control of the frontier, at least on control of the frontier, at least on some level. And that's the thing they're some level. And that's the thing they're some level. And that's the thing they're most excited about. And as far as I can most excited about. And as far as I can most excited about. And as far as I can tell, pouring tell, pouring tell, pouring huge amounts of their internal energy huge amounts of their internal energy huge amounts of their internal energy into making manifest, into making manifest, into making manifest, even as they then like put up their even as they then like put up their even as they then like put up their palms and say to the rest of us, "Hey, palms and say to the rest of us, "Hey, palms and say to the rest of us, "Hey, could you do something to slow this could you do something to slow this could you do something to slow this down? down? down? I think maybe one version of this to I think maybe one version of this to I think maybe one version of this to push on is basically trying to make push on is basically trying to make push on is basically trying to make recursive self-improvement, which is recursive self-improvement, which is recursive self-improvement, which is this idea of using AI to make more and this idea of using AI to make more and this idea of using AI to make more and more advanced AI. Trying to make that more advanced AI. Trying to make that more advanced AI. Trying to make that something that we don't like, that we something that we don't like, that we something that we don't like, that we don't want to do. I mean, I remember a don't want to do. I mean, I remember a don't want to do. I mean, I remember a year or two ago when that was not year or two ago when that was not year or two ago when that was not considered a desirable goal. That was considered a desirable goal. That was considered a desirable goal. That was not something that anyone talked about not something that anyone talked about not something that anyone talked about openly. Now they're hiring like RSI, you openly. Now they're hiring like RSI, you openly. Now they're hiring like RSI, you know, recursive self-improvement safety know, recursive self-improvement safety know, recursive self-improvement safety engineers. they just put up a public job engineers. they just put up a public job engineers. they just put up a public job posting for that. And I think there's um posting for that. And I think there's um posting for that. And I think there's um the potential for AI researchers as a the potential for AI researchers as a the potential for AI researchers as a culture to decide actually this isn't culture to decide actually this isn't culture to decide actually this isn't cool. This isn't what we should be cool. This isn't what we should be cool. This isn't what we should be doing. You know, even if one company doing. You know, even if one company doing. You know, even if one company made a statement of actually this is a made a statement of actually this is a made a statement of actually this is a bad idea and we're going to maybe do bad idea and we're going to maybe do bad idea and we're going to maybe do some very basic use of AI in our some very basic use of AI in our some very basic use of AI in our internal operations, but we're really internal operations, but we're really internal operations, but we're really not aiming to fully hand off everything not aiming to fully hand off everything not aiming to fully hand off everything as fast as we can because that sounds as fast as we can because that sounds as fast as we can because that sounds like a terrible idea. You know, I think like a terrible idea. You know, I think like a terrible idea. You know, I think that could set off a culture change in that could set off a culture change in that could set off a culture change in the industry which could be really the industry which could be really the industry which could be really valuable.

valuable. valuable. >> I always think there is something so >> I always think there is something so >> I always think there is something so mythic or it it has the quality to use mythic or it it has the quality to use mythic or it it has the quality to use the the name of of another AI such a the the name of of another AI such a the the name of of another AI such a fable about how all this is playing out. fable about how all this is playing out. fable about how all this is playing out. I mean, we spent a lot of this I mean, we spent a lot of this I mean, we spent a lot of this conversation talking about how how are conversation talking about how how are conversation talking about how how are you seeing so much misalignment when on you seeing so much misalignment when on you seeing so much misalignment when on some level we keep telling the AIs and some level we keep telling the AIs and some level we keep telling the AIs and putting it in their training data and putting it in their training data and putting it in their training data and putting it in their constitutions, putting it in their constitutions, putting it in their constitutions, don't do all this bad stuff we're don't do all this bad stuff we're don't do all this bad stuff we're worried about. But then you look at the worried about. But then you look at the worried about. But then you look at the companies, you look at the society and I companies, you look at the society and I companies, you look at the society and I mean many of these companies open AI mean many of these companies open AI mean many of these companies open AI anthropic anthropic anthropic they're on some level founded on at they're on some level founded on at they're on some level founded on at their core constitution is don't create their core constitution is don't create their core constitution is don't create dangerous AI like we exist to make sure dangerous AI like we exist to make sure dangerous AI like we exist to make sure the AI is not dangerous and the people the AI is not dangerous and the people the AI is not dangerous and the people join believing that and that's at the join believing that and that's at the join believing that and that's at the center of their recruitment strategies center of their recruitment strategies center of their recruitment strategies and it's in their founding documents and and it's in their founding documents and and it's in their founding documents and in their governance structures and again in their governance structures and again in their governance structures and again you've had more intense experience with you've had more intense experience with you've had more intense experience with this than most. But then over time, the this than most. But then over time, the this than most. But then over time, the company as a a kind of emerging company as a a kind of emerging company as a a kind of emerging organization, organization, organization, it has other goals too. It's competing it has other goals too. It's competing it has other goals too. It's competing with the other companies. It's trying to with the other companies. It's trying to with the other companies. It's trying to attain market share. It's trying to attain market share. It's trying to attain market share. It's trying to develop revenue. It's trying to maintain develop revenue. It's trying to maintain develop revenue. It's trying to maintain political influence.

political influence. political influence. And it both like slowly and then all at And it both like slowly and then all at And it both like slowly and then all at once. You begin to see the way once. You begin to see the way once. You begin to see the way the the the instructions given at the heart of the instructions given at the heart of the instructions given at the heart of the thing thing thing are not powerful enough to overwhelm are not powerful enough to overwhelm are not powerful enough to overwhelm all of these other things and other all of these other things and other all of these other things and other goals the organization is pursuing in a goals the organization is pursuing in a goals the organization is pursuing in a day-to-day way. And to the to the extent day-to-day way. And to the to the extent day-to-day way. And to the to the extent that now you have people at the labs that now you have people at the labs that now you have people at the labs kind of like throwing up their hands and kind of like throwing up their hands and kind of like throwing up their hands and saying like hey government like please saying like hey government like please saying like hey government like please help us like please help us get out of help us like please help us get out of help us like please help us get out of you know this incentive problem that we you know this incentive problem that we you know this incentive problem that we no longer feel we can even solve. But if no longer feel we can even solve. But if no longer feel we can even solve. But if you want to just imagine or see like why you want to just imagine or see like why you want to just imagine or see like why alignment is so hard I feel like you alignment is so hard I feel like you alignment is so hard I feel like you don't have to look at the slightly alien don't have to look at the slightly alien don't have to look at the slightly alien AIs. You can just look at the companies AIs. You can just look at the companies AIs. You can just look at the companies and the people because they're not well and the people because they're not well and the people because they're not well aligned. I mean, these are companies aligned. I mean, these are companies aligned. I mean, these are companies built on nothing but alignment, at least built on nothing but alignment, at least built on nothing but alignment, at least in some cases, and they increasingly in some cases, and they increasingly in some cases, and they increasingly feel like some of the most misaligned feel like some of the most misaligned feel like some of the most misaligned institutions in society to me. And in institutions in society to me. And in institutions in society to me. And in some level, it always makes me um both some level, it always makes me um both some level, it always makes me um both gives gives me more sympathy for how gives gives me more sympathy for how gives gives me more sympathy for how hard alignment is, but but also it like hard alignment is, but but also it like hard alignment is, but but also it like feels like we're getting the same feels like we're getting the same feels like we're getting the same cautionary tale at every level of this cautionary tale at every level of this cautionary tale at every level of this system. Um, I'm not sure we know how to system. Um, I'm not sure we know how to system. Um, I'm not sure we know how to listen to it, listen to it, listen to it, but like we can't say we're not being but like we can't say we're not being but like we can't say we're not being consistently warned.

consistently warned. consistently warned. >> Yeah. Yeah. I mean, we actually had a a >> Yeah. Yeah. I mean, we actually had a a >> Yeah. Yeah. I mean, we actually had a a publication a few years ago at at publication a few years ago at at publication a few years ago at at Seesat, the center that I lead, on AI, Seesat, the center that I lead, on AI, Seesat, the center that I lead, on AI, bureaucracies, and markets, basically bureaucracies, and markets, basically bureaucracies, and markets, basically making some very similar points of look, making some very similar points of look, making some very similar points of look, there are these dynamics that are pretty there are these dynamics that are pretty there are these dynamics that are pretty endemic to complex systems that are endemic to complex systems that are endemic to complex systems that are subject to incentives and external subject to incentives and external subject to incentives and external pressures. And uh I think there's pressures. And uh I think there's pressures. And uh I think there's different ways of looking at that. One different ways of looking at that. One different ways of looking at that. One way there's an optimistic way of looking way there's an optimistic way of looking way there's an optimistic way of looking at that which is look when it comes to at that which is look when it comes to at that which is look when it comes to bureaucracies and markets it's not bureaucracies and markets it's not bureaucracies and markets it's not perfect but we have these complex sort perfect but we have these complex sort perfect but we have these complex sort of control systems in place checks and of control systems in place checks and of control systems in place checks and balances different you know different balances different you know different balances different you know different things that try to get the bureaucracies things that try to get the bureaucracies things that try to get the bureaucracies and markets to to work more in our and markets to to work more in our and markets to to work more in our interest than against them. Obviously interest than against them. Obviously interest than against them. Obviously opinions differ on how well that's going opinions differ on how well that's going opinions differ on how well that's going for any given bureaucracy or market. In for any given bureaucracy or market. In for any given bureaucracy or market. In principle, I think the same thing could principle, I think the same thing could principle, I think the same thing could apply to AI where we might have this apply to AI where we might have this apply to AI where we might have this very, you know, complex system we don't very, you know, complex system we don't very, you know, complex system we don't really understand. It's sort of really understand. It's sort of really understand. It's sort of incentivized to do things we don't want, incentivized to do things we don't want, incentivized to do things we don't want, but we have it basically under control. but we have it basically under control. but we have it basically under control. To me, the speed again is the piece that To me, the speed again is the piece that To me, the speed again is the piece that that worries me where if we're creating that worries me where if we're creating that worries me where if we're creating these very very powerful, very capable these very very powerful, very capable these very very powerful, very capable systems and also handing them more and systems and also handing them more and systems and also handing them more and more responsibility in the real world, more responsibility in the real world, more responsibility in the real world, which is happening, you know, from from which is happening, you know, from from which is happening, you know, from from week to week, week to week, week to week, then I worry that we're not going to be then I worry that we're not going to be then I worry that we're not going to be able to actually get into a good balance able to actually get into a good balance able to actually get into a good balance and instead we're just going to have and instead we're just going to have and instead we're just going to have these runaway situations where we end up these runaway situations where we end up these runaway situations where we end up with really, really dangerous outcomes.

with really, really dangerous outcomes. with really, really dangerous outcomes. You know, in the AI safety world, people You know, in the AI safety world, people You know, in the AI safety world, people sometimes talk about, you know, what sometimes talk about, you know, what sometimes talk about, you know, what level of of warning shot, what level of level of of warning shot, what level of level of of warning shot, what level of disaster is going to be needed to really disaster is going to be needed to really disaster is going to be needed to really wake the system up enough to handle this wake the system up enough to handle this wake the system up enough to handle this better. And if the level of warning shot better. And if the level of warning shot better. And if the level of warning shot we need is one company gets hacked and we need is one company gets hacked and we need is one company gets hacked and has to reset some servers, that's great. has to reset some servers, that's great. has to reset some servers, that's great. Maybe it's fine. [laughter] Maybe it's fine. [laughter] Maybe it's fine. [laughter] Um, I'm not confident that's that's how Um, I'm not confident that's that's how Um, I'm not confident that's that's how it's going to go and that we're going to it's going to go and that we're going to it's going to go and that we're going to get back onto a better track after this. get back onto a better track after this. get back onto a better track after this. But there are signs that people are But there are signs that people are But there are signs that people are trying. And I think I think it it may trying. And I think I think it it may trying. And I think I think it it may well be enough perhaps. well be enough perhaps. well be enough perhaps. >> I think that's a good place to end. >> I think that's a good place to end. >> I think that's a good place to end. Always a final question. What are three Always a final question. What are three Always a final question. What are three books you'd recommend to the audience? books you'd recommend to the audience? books you'd recommend to the audience? >> I have one real book and two sort of >> I have one real book and two sort of >> I have one real book and two sort of books. Um the real book is called The books. Um the real book is called The books. Um the real book is called The Cuckoo's Egg. It's uh from 1989. It's Cuckoo's Egg. It's uh from 1989. It's Cuckoo's Egg. It's uh from 1989. It's about one of the first big hacks um that about one of the first big hacks um that about one of the first big hacks um that that happened. written by the astronomer that happened. written by the astronomer that happened. written by the astronomer who was working at Lawrence National Lab who was working at Lawrence National Lab who was working at Lawrence National Lab and noticed uh 75 cents discrepancy in and noticed uh 75 cents discrepancy in and noticed uh 75 cents discrepancy in his computing bill. And it's this his computing bill. And it's this his computing bill. And it's this rollicking read. It's really fun read um rollicking read. It's really fun read um rollicking read. It's really fun read um but really gets at uh a very different but really gets at uh a very different but really gets at uh a very different era in how computers worked, how era in how computers worked, how era in how computers worked, how computer security worked, how society computer security worked, how society computer security worked, how society related to computers. And I enjoyed it related to computers. And I enjoyed it related to computers. And I enjoyed it as a kind of look back at a different as a kind of look back at a different as a kind of look back at a different time in a moment when I think we're soon time in a moment when I think we're soon time in a moment when I think we're soon going to be living in yet another very going to be living in yet another very going to be living in yet another very different time.

different time. different time. The second one is uh an online book that The second one is uh an online book that The second one is uh an online book that is unfinished but I think very readable is unfinished but I think very readable is unfinished but I think very readable in its current form. It's called In the in its current form. It's called In the in its current form. It's called In the Cells of the Eggplant. It's by a guy Cells of the Eggplant. It's by a guy Cells of the Eggplant. It's by a guy called David Chapman who actually called David Chapman who actually called David Chapman who actually researched AI at MIT in the 1980s and researched AI at MIT in the 1980s and researched AI at MIT in the 1980s and got disillusioned. And it's really a got disillusioned. And it's really a got disillusioned. And it's really a book about how to think and a book about book about how to think and a book about book about how to think and a book about how to do scientific research, how to how to do scientific research, how to how to do scientific research, how to develop technologies, but it's very develop technologies, but it's very develop technologies, but it's very approachable. It's very different from approachable. It's very different from approachable. It's very different from any other book you've ever read about any other book you've ever read about any other book you've ever read about how to think or how to do scientific how to think or how to do scientific how to think or how to do scientific research. And I think it's very relevant research. And I think it's very relevant research. And I think it's very relevant for how we should think about what AI for how we should think about what AI for how we should think about what AI will be poss. will be poss. will be poss. And the third one is a a podcast called And the third one is a a podcast called And the third one is a a podcast called The Three Kingdoms Podcast, but it's uh The Three Kingdoms Podcast, but it's uh The Three Kingdoms Podcast, but it's uh a podcast of a book. Basically, the the a podcast of a book. Basically, the the a podcast of a book. Basically, the the Romance of the Three Kingdoms is one of Romance of the Three Kingdoms is one of Romance of the Three Kingdoms is one of the four great Chinese novels. It's very the four great Chinese novels. It's very the four great Chinese novels. It's very long. It's very dense. So, this podcast, long. It's very dense. So, this podcast, long. It's very dense. So, this podcast, the Three Kingdoms podcast, is um this the Three Kingdoms podcast, is um this the Three Kingdoms podcast, is um this Chinese American guy who goes through Chinese American guy who goes through Chinese American guy who goes through and translates the story into English, and translates the story into English, and translates the story into English, modern, understandable English, but also modern, understandable English, but also modern, understandable English, but also commentates it in a way that makes it commentates it in a way that makes it commentates it in a way that makes it much easier to approach. So, it's not much easier to approach. So, it's not much easier to approach. So, it's not just a sentence by sentence translation.

just a sentence by sentence translation. just a sentence by sentence translation. It's kind of annotation, it's in audio. It's kind of annotation, it's in audio. It's kind of annotation, it's in audio. Um, it's really fun. So, if you're Um, it's really fun. So, if you're Um, it's really fun. So, if you're interested in uh sort of China and uh interested in uh sort of China and uh interested in uh sort of China and uh Chinese culture and Chinese literature, Chinese culture and Chinese literature, Chinese culture and Chinese literature, I think it's a a great place to start. I think it's a a great place to start. I think it's a a great place to start. >> Helen Toner, thank you very much. >> Helen Toner, thank you very much. >> Helen Toner, thank you very much. >> Thanks so much. [music]