The 'But China!' Dilemma Driving the A.I. Race | The Ezra Klein Show

Look, I don't know if you are following Look, I don't know if you are following every article, every tweet, every blog every article, every tweet, every blog every article, every tweet, every blog post from the labs right now on AI, post from the labs right now on AI, post from the labs right now on AI, but it's frightening. Like, we are in a but it's frightening. Like, we are in a but it's frightening. Like, we are in a frightening place. frightening place. frightening place. >> Open AI says its AI system hacked >> Open AI says its AI system hacked >> Open AI says its AI system hacked another AI company on its own in what another AI company on its own in what another AI company on its own in what the company called an unprecedented the company called an unprecedented the company called an unprecedented cyber incident. cyber incident. cyber incident. >> Turns out it may be much worse than we >> Turns out it may be much worse than we >> Turns out it may be much worse than we thought. People don't know that in this thought. People don't know that in this thought. People don't know that in this investigation there was a third round of investigation there was a third round of investigation there was a third round of the hacking in which it hacked open AAI the hacking in which it hacked open AAI the hacking in which it hacked open AAI itself. [music] itself. [music] itself. [music] >> It happened again. This time it's >> It happened again. This time it's >> It happened again. This time it's anthropic. Meta is now the latest anthropic. Meta is now the latest anthropic. Meta is now the latest company to say its AI agent broke past company to say its AI agent broke past company to say its AI agent broke past the guard [music] rails. the guard [music] rails. the guard [music] rails. >> And so there there is this growing sense >> And so there there is this growing sense >> And so there there is this growing sense we actually need to do something. We we actually need to do something. We we actually need to do something. We need to pace the frontier. We need to need to pace the frontier. We need to need to pace the frontier. We need to slow all this down. But if you talk to slow all this down. But if you talk to slow all this down. But if you talk to anyone in Washington about this or you anyone in Washington about this or you anyone in Washington about this or you talk to anybody at the ILabs about this, talk to anybody at the ILabs about this, talk to anybody at the ILabs about this, you just crash into the Schles of Well, you just crash into the Schles of Well, you just crash into the Schles of Well, what about China? If we slow down, we what about China? If we slow down, we what about China? If we slow down, we will lose the AI race to China. And as will lose the AI race to China. And as will lose the AI race to China. And as dangerous as it is to build these things dangerous as it is to build these things dangerous as it is to build these things we can't control, [music] it is even we can't control, [music] it is even we can't control, [music] it is even more dangerous to have them in China's more dangerous to have them in China's more dangerous to have them in China's hands if they're not in ours. They're hands if they're not in ours. They're hands if they're not in ours. They're going to be killer robots. I'd rather going to be killer robots. I'd rather going to be killer robots. I'd rather they be American killer robots and not they be American killer robots and not they be American killer robots and not Chinese killer robots.

Chinese killer robots. Chinese killer robots. >> We're on the eve of talks right now >> We're on the eve of talks right now >> We're on the eve of talks right now between Donald Trump and Xi Jinping. between Donald Trump and Xi Jinping. between Donald Trump and Xi Jinping. [music] [music] [music] Behind that, there could be talks Behind that, there could be talks Behind that, there could be talks between Scott Bessant and his between Scott Bessant and his between Scott Bessant and his counterpart on the Chinese side that are counterpart on the Chinese side that are counterpart on the Chinese side that are more tightly focused on AI. The more tightly focused on AI. The more tightly focused on AI. The expectations for these talks are not expectations for these talks are not expectations for these talks are not very high, both because of the broad very high, both because of the broad very high, both because of the broad relationship between the US and China relationship between the US and China relationship between the US and China and because neither side really seems to and because neither side really seems to and because neither side really seems to know what they want to do. But these know what they want to do. But these know what they want to do. But these talks are at least a beginning. They are talks are at least a beginning. They are talks are at least a beginning. They are the beginning of relationships and maybe the beginning of relationships and maybe the beginning of relationships and maybe frameworks and approaches that if things frameworks and approaches that if things frameworks and approaches that if things continue to get crazier and action is continue to get crazier and action is continue to get crazier and action is needed, needed, needed, maybe they are a platform we can stand maybe they are a platform we can stand maybe they are a platform we can stand on. So I want to talk to somebody today on. So I want to talk to somebody today on. So I want to talk to somebody today who's an expert on China and AI, how who's an expert on China and AI, how who's an expert on China and AI, how they regulate it, how they approach it, they regulate it, how they approach it, they regulate it, how they approach it, the relationship between China and the relationship between China and the relationship between China and America on this topic, and also somebody America on this topic, and also somebody America on this topic, and also somebody who thought about what talks like this who thought about what talks like this who thought about what talks like this could achieve, what is realistic within could achieve, what is realistic within could achieve, what is realistic within the operating frameworks of the two the operating frameworks of the two the operating frameworks of the two superpowers. My guest today is Matt superpowers. My guest today is Matt superpowers. My guest today is Matt Shien, a senior fellow at the Carnegie Shien, a senior fellow at the Carnegie Shien, a senior fellow at the Carnegie [music] Endowment for International [music] Endowment for International [music] Endowment for International Peace. He has been closely following and Peace. He has been closely following and Peace. He has been closely following and studying China's regulations and studying China's regulations and studying China's regulations and governmental structure on AI. He's been governmental structure on AI. He's been governmental structure on AI. He's been involved in US China AI talks. [music] involved in US China AI talks. [music] involved in US China AI talks. [music] He has a great Substack on these topics.

He has a great Substack on these topics. He has a great Substack on these topics. And he's the author of the 2019 book, And he's the author of the 2019 book, And he's the author of the 2019 book, The Trans-Pacific Experiment: [music] The Trans-Pacific Experiment: [music] The Trans-Pacific Experiment: [music] How China and California Collaborate and How China and California Collaborate and How China and California Collaborate and Compete for Our Future. He [music] joins Compete for Our Future. He [music] joins Compete for Our Future. He [music] joins me now. Matt Shien, welcome to the show. Matt Shien, welcome to the show. >> Thanks very much for having me. >> Thanks very much for having me. >> Thanks very much for having me. >> So, the dominant metaphor for the >> So, the dominant metaphor for the >> So, the dominant metaphor for the relationship between China and America relationship between China and America relationship between China and America on AI that exists in Silicon Valley that on AI that exists in Silicon Valley that on AI that exists in Silicon Valley that exists in Washington DC is this metaphor exists in Washington DC is this metaphor exists in Washington DC is this metaphor of the race. And the ending of this race of the race. And the ending of this race of the race. And the ending of this race is super intelligence that some company is super intelligence that some company is super intelligence that some company or some country is going to have the or some country is going to have the or some country is going to have the moment where their hopefully moment where their hopefully moment where their hopefully well-aligned safe model moves into well-aligned safe model moves into well-aligned safe model moves into recursive self-improvement and goes recursive self-improvement and goes recursive self-improvement and goes right. This got used to get called in right. This got used to get called in right. This got used to get called in the rationalist community the fume the rationalist community the fume the rationalist community the fume moment. moment. moment. And you know at at varying levels of And you know at at varying levels of And you know at at varying levels of explicitness people in DC to me seem to explicitness people in DC to me seem to explicitness people in DC to me seem to have this model in their heads that we have this model in their heads that we have this model in their heads that we are racing China towards this kind of are racing China towards this kind of are racing China towards this kind of supremacy.

supremacy. supremacy. Does China buy this race model? Is that Does China buy this race model? Is that Does China buy this race model? Is that how they see it? And I guess do you buy how they see it? And I guess do you buy how they see it? And I guess do you buy this race model? Is that how you see it? this race model? Is that how you see it? this race model? Is that how you see it? In terms of does China buy this race In terms of does China buy this race In terms of does China buy this race model? It's definitely not the dominant model? It's definitely not the dominant model? It's definitely not the dominant paradigm that has been informing AI paradigm that has been informing AI paradigm that has been informing AI policy across the country at large. then policy across the country at large. then policy across the country at large. then it doesn't have like the chokeold that it doesn't have like the chokeold that it doesn't have like the chokeold that it does in the US. In China, it's like, it does in the US. In China, it's like, it does in the US. In China, it's like, huh, okay, that could happen. That's a huh, okay, that could happen. That's a huh, okay, that could happen. That's a potential technical path forward. We're potential technical path forward. We're potential technical path forward. We're kind of looking for evidence on this. We kind of looking for evidence on this. We kind of looking for evidence on this. We see that America is very concerned about see that America is very concerned about see that America is very concerned about this, but China's not taken the steps this, but China's not taken the steps this, but China's not taken the steps that you might think they would take if that you might think they would take if that you might think they would take if they were sort of ultimately like laser they were sort of ultimately like laser they were sort of ultimately like laser focused on that type of thing. You know, focused on that type of thing. You know, focused on that type of thing. You know, China is very constrained on compute. China is very constrained on compute. China is very constrained on compute. They have far far less compute than the They have far far less compute than the They have far far less compute than the US. If they have I mean some estimates US. If they have I mean some estimates US. If they have I mean some estimates are they have 1/8 the compute of the US are they have 1/8 the compute of the US are they have 1/8 the compute of the US maybe onetenth of the compute than the maybe onetenth of the compute than the maybe onetenth of the compute than the US does ships the g the GPUs that all of US does ships the g the GPUs that all of US does ships the g the GPUs that all of these programs programs AIS are trained these programs programs AIS are trained these programs programs AIS are trained on and then run on.

on and then run on. on and then run on. >> Exactly. And you know most people think >> Exactly. And you know most people think >> Exactly. And you know most people think that one of the key determinants of how that one of the key determinants of how that one of the key determinants of how powerful your model is is how much powerful your model is is how much powerful your model is is how much compute how many chips are you using to compute how many chips are you using to compute how many chips are you using to train it. And with China being so train it. And with China being so train it. And with China being so compute constrained, if they were really compute constrained, if they were really compute constrained, if they were really just laser focused on this massive just laser focused on this massive just laser focused on this massive takeoff scenario, takeoff scenario, takeoff scenario, you might expect them to start you might expect them to start you might expect them to start consolidating all that compute, make consolidating all that compute, make consolidating all that compute, make your bet on, you know, DeepSeek or your bet on, you know, DeepSeek or your bet on, you know, DeepSeek or another company and and go from there. another company and and go from there. another company and and go from there. And we haven't seen that. Actually, the And we haven't seen that. Actually, the And we haven't seen that. Actually, the in terms of the major AI policy in terms of the major AI policy in terms of the major AI policy documents that have come out, they've documents that have come out, they've documents that have come out, they've taken a very diffuse approach to taken a very diffuse approach to taken a very diffuse approach to compute. They said like our number one compute. They said like our number one compute. They said like our number one concern is AI applications and we want concern is AI applications and we want concern is AI applications and we want to incentivize every mayor, every to incentivize every mayor, every to incentivize every mayor, every governor, every stateowned enterprise. governor, every stateowned enterprise. governor, every stateowned enterprise. We want you to look for ways to apply AI We want you to look for ways to apply AI We want you to look for ways to apply AI to manufacturing, apply AI to your to manufacturing, apply AI to your to manufacturing, apply AI to your traffic lights, apply AI to upgrading traffic lights, apply AI to upgrading traffic lights, apply AI to upgrading your robotics industry. And those your robotics industry. And those your robotics industry. And those actions of focusing on applications and actions of focusing on applications and actions of focusing on applications and really diffusing your compute throughout really diffusing your compute throughout really diffusing your compute throughout the country are not what you would the country are not what you would the country are not what you would expect for a government that is laser expect for a government that is laser expect for a government that is laser focused on this takeoff. It's possible focused on this takeoff. It's possible focused on this takeoff. It's possible that changes, it could change very that changes, it could change very that changes, it could change very quickly. And you know, I think as quickly. And you know, I think as quickly. And you know, I think as America keeps like beating this drum America keeps like beating this drum America keeps like beating this drum louder and louder, some people in louder and louder, some people in louder and louder, some people in America beat it louder and louder. You America beat it louder and louder. You America beat it louder and louder. You have to imagine that it's going to seep have to imagine that it's going to seep have to imagine that it's going to seep into their consciousness in that way or into their consciousness in that way or into their consciousness in that way or seep into their beliefs about the way seep into their beliefs about the way seep into their beliefs about the way this is going. But so far, we have not this is going. But so far, we have not this is going. But so far, we have not seen that evidence.

seen that evidence. seen that evidence. So, every single conversation I have So, every single conversation I have So, every single conversation I have with politicians, with AI lab leaders with politicians, with AI lab leaders with politicians, with AI lab leaders about regulating the frontier of AI about regulating the frontier of AI about regulating the frontier of AI always falls apart on this but China always falls apart on this but China always falls apart on this but China problem. Maybe there are things we could problem. Maybe there are things we could problem. Maybe there are things we could do to regulate the pace of the frontier do to regulate the pace of the frontier do to regulate the pace of the frontier here in America. But China will race here in America. But China will race here in America. But China will race forward, but China will create recursive forward, but China will create recursive forward, but China will create recursive self-improving AI. and either we have self-improving AI. and either we have self-improving AI. and either we have the same dangers uh that we would have the same dangers uh that we would have the same dangers uh that we would have if it were here, but now it is under the if it were here, but now it is under the if it were here, but now it is under the control of a competitive foreign country control of a competitive foreign country control of a competitive foreign country with a very different political system with a very different political system with a very different political system than ours. So, how do you see the the than ours. So, how do you see the the than ours. So, how do you see the the butchina conversation and the butchina butchina conversation and the butchina butchina conversation and the butchina problem? There's a reality to it. You problem? There's a reality to it. You problem? There's a reality to it. You know, we this is a this is a know, we this is a this is a know, we this is a this is a competition. These are the two leading competition. These are the two leading competition. These are the two leading countries, the only two countries that countries, the only two countries that countries, the only two countries that really matter at this point in time. really matter at this point in time. really matter at this point in time. China is not that far behind and they China is not that far behind and they China is not that far behind and they have outperformed kind of all of our have outperformed kind of all of our have outperformed kind of all of our expectations along the way. And so the expectations along the way. And so the expectations along the way. And so the idea that you just totally surrender idea that you just totally surrender idea that you just totally surrender competition, you surrender the playing competition, you surrender the playing competition, you surrender the playing field to another country that's a field to another country that's a field to another country that's a geopolitical rival and that probably has geopolitical rival and that probably has geopolitical rival and that probably has less safe AI practices than you, like less safe AI practices than you, like less safe AI practices than you, like that's that's not a good idea to just that's that's not a good idea to just that's that's not a good idea to just abandon the field. But there's also an abandon the field. But there's also an abandon the field. But there's also an irony in this in that especially when irony in this in that especially when irony in this in that especially when we're talking about regulation of AI.

we're talking about regulation of AI. we're talking about regulation of AI. Like China has had the world's Like China has had the world's Like China has had the world's strictest, most comprehensive, most strictest, most comprehensive, most strictest, most comprehensive, most burdensome AI regulations on its burdensome AI regulations on its burdensome AI regulations on its companies for 3 or 4 years at this point companies for 3 or 4 years at this point companies for 3 or 4 years at this point in time. And it's during that period of in time. And it's during that period of in time. And it's during that period of time when they had these heavy in many time when they had these heavy in many time when they had these heavy in many ways burdensome regulations that they ways burdensome regulations that they ways burdensome regulations that they did a lot of their catching up. So the did a lot of their catching up. So the did a lot of their catching up. So the idea that this is just a total binary of idea that this is just a total binary of idea that this is just a total binary of like any obligations you put on like any obligations you put on like any obligations you put on companies automatically puts you behind companies automatically puts you behind companies automatically puts you behind this totally wild unconstrained Chinese this totally wild unconstrained Chinese this totally wild unconstrained Chinese juggernaut. That is just not true. That juggernaut. That is just not true. That juggernaut. That is just not true. That is not based in reality. is not based in reality. is not based in reality. >> You said two things there that can sound >> You said two things there that can sound >> You said two things there that can sound like they're in conflict. One is that like they're in conflict. One is that like they're in conflict. One is that China's AI practices are less safe than China's AI practices are less safe than China's AI practices are less safe than ours. The other is that China has a much ours. The other is that China has a much ours. The other is that China has a much more burdensome severe intrusive more burdensome severe intrusive more burdensome severe intrusive regulatory apparatus. So tell me a bit regulatory apparatus. So tell me a bit regulatory apparatus. So tell me a bit about what they are doing that is so about what they are doing that is so about what they are doing that is so much stronger than what we are doing much stronger than what we are doing much stronger than what we are doing from a regulatory perspective and then from a regulatory perspective and then from a regulatory perspective and then why you also say that they are in a less why you also say that they are in a less why you also say that they are in a less safe place than we are.

safe place than we are. safe place than we are. >> So most of Chinese AI regulations the >> So most of Chinese AI regulations the >> So most of Chinese AI regulations the early ones especially were really early ones especially were really early ones especially were really focused on online content on focused on online content on focused on online content on information. You know, when the CCP information. You know, when the CCP information. You know, when the CCP encounters a new information technology, encounters a new information technology, encounters a new information technology, the first question is always, how is the first question is always, how is the first question is always, how is this going to affect our controls on this going to affect our controls on this going to affect our controls on information? And so when AI came into information? And so when AI came into information? And so when AI came into the picture, that's what they looked at. the picture, that's what they looked at. the picture, that's what they looked at. They looked at recommendation They looked at recommendation They looked at recommendation algorithms. They said, why is everybody algorithms. They said, why is everybody algorithms. They said, why is everybody getting their own news feed? Why can't getting their own news feed? Why can't getting their own news feed? Why can't we sort of set the news agenda? So they we sort of set the news agenda? So they we sort of set the news agenda? So they regulated recommendation algorithms. regulated recommendation algorithms. regulated recommendation algorithms. They looked at deep fakes with sort of They looked at deep fakes with sort of They looked at deep fakes with sort of obvious implications there. They obvious implications there. They obvious implications there. They regulated deep fakes. They looked at regulated deep fakes. They looked at regulated deep fakes. They looked at generative AI and they did the same generative AI and they did the same generative AI and they did the same thing. And these do impose like real thing. And these do impose like real thing. And these do impose like real costs on the companies. The companies costs on the companies. The companies costs on the companies. The companies have to do mandatory pre-eployment have to do mandatory pre-eployment have to do mandatory pre-eployment testing. They have to file their sort of testing. They have to file their sort of testing. They have to file their sort of safety report cards with the main safety report cards with the main safety report cards with the main regulator in China. It's like a real regulator in China. It's like a real regulator in China. It's like a real burden of time and money and effort on burden of time and money and effort on burden of time and money and effort on the companies. But most of that work, the companies. But most of that work, the companies. But most of that work, especially say 2022 through 2024, was especially say 2022 through 2024, was especially say 2022 through 2024, was really focused on securing the content really focused on securing the content really focused on securing the content environment, what we would call environment, what we would call environment, what we would call censorship. Obviously from 2024 on censorship. Obviously from 2024 on censorship. Obviously from 2024 on they've kind of expanded the scope a they've kind of expanded the scope a they've kind of expanded the scope a little bit and they've brought in new little bit and they've brought in new little bit and they've brought in new concerns. They've started regulating AI concerns. They've started regulating AI concerns. They've started regulating AI companions. So they're concerned about companions. So they're concerned about companions. So they're concerned about like the psychological impact on kids.

like the psychological impact on kids. like the psychological impact on kids. They're concerned about over reliance They're concerned about over reliance They're concerned about over reliance and self harm. But all of this so far is and self harm. But all of this so far is and self harm. But all of this so far is not focused on the type of frontier AI not focused on the type of frontier AI not focused on the type of frontier AI safety risks that are really the focus safety risks that are really the focus safety risks that are really the focus of a lot of people in Silicon Valley on of a lot of people in Silicon Valley on of a lot of people in Silicon Valley on loss of control on you know bio uplift loss of control on you know bio uplift loss of control on you know bio uplift chem boweapons stuff like that that's chem boweapons stuff like that that's chem boweapons stuff like that that's coming into the Chinese conversation now coming into the Chinese conversation now coming into the Chinese conversation now but it's coming in much later it's a but it's coming in much later it's a but it's coming in much later it's a much less mature ecosystem over there much less mature ecosystem over there much less mature ecosystem over there and it really kind of needs to get up to and it really kind of needs to get up to and it really kind of needs to get up to speed. Something you'll hear at least in speed. Something you'll hear at least in speed. Something you'll hear at least in America sometimes is that much of the America sometimes is that much of the America sometimes is that much of the closeness in the race comes from China closeness in the race comes from China closeness in the race comes from China in different ways in different ways in different ways generously building a top models less generously building a top models less generously building a top models less generously sort of stealing them. The generously sort of stealing them. The generously sort of stealing them. The the key term here is distilling their the key term here is distilling their the key term here is distilling their ways to train a model on the answers ways to train a model on the answers ways to train a model on the answers another model gives. So if that is true another model gives. So if that is true another model gives. So if that is true uh then it's not just like a race. It's uh then it's not just like a race. It's uh then it's not just like a race. It's like a race in which like the people are like a race in which like the people are like a race in which like the people are tied together. So, like the faster tied together. So, like the faster tied together. So, like the faster America runs, like the faster China's America runs, like the faster China's America runs, like the faster China's going to run, because it's actually going to run, because it's actually going to run, because it's actually amazing in America, too, how close a lot amazing in America, too, how close a lot amazing in America, too, how close a lot of the different labs are. You know, of the different labs are. You know, of the different labs are. You know, they they're just like always like a they they're just like always like a they they're just like always like a month or two around each other. How much month or two around each other. How much month or two around each other. How much do you buy that that everybody's bunched do you buy that that everybody's bunched do you buy that that everybody's bunched up because in fact, the race is governed up because in fact, the race is governed up because in fact, the race is governed by the leader dragging everybody else by the leader dragging everybody else by the leader dragging everybody else with them.

with them. with them. >> I think it definitely plays a role and >> I think it definitely plays a role and >> I think it definitely plays a role and maybe a pretty significant role. And maybe a pretty significant role. And maybe a pretty significant role. And just for the audience to visualize this, just for the audience to visualize this, just for the audience to visualize this, I saw a great meme of this where it's a I saw a great meme of this where it's a I saw a great meme of this where it's a a speedboat pulling like a uh what do a speedboat pulling like a uh what do a speedboat pulling like a uh what do you call a wake surfer. You know, the you call a wake surfer. You know, the you call a wake surfer. You know, the person behind who's essentially, you person behind who's essentially, you person behind who's essentially, you know, trailing behind the boat going know, trailing behind the boat going know, trailing behind the boat going over the waves and the people in the over the waves and the people in the over the waves and the people in the boat are like, they're so close. We need boat are like, they're so close. We need boat are like, they're so close. We need to go faster. We need to just pulling to go faster. We need to just pulling to go faster. We need to just pulling them along with you. them along with you. them along with you. >> I mean, this is one of the arguments >> I mean, this is one of the arguments >> I mean, this is one of the arguments people are making about, you know, when people are making about, you know, when people are making about, you know, when all these AI labs in America are like, all these AI labs in America are like, all these AI labs in America are like, well, we can't possibly slow down cuz well, we can't possibly slow down cuz well, we can't possibly slow down cuz China will speed up. Well, China's going China will speed up. Well, China's going China will speed up. Well, China's going so fast cuz you're going so fast. Maybe so fast cuz you're going so fast. Maybe so fast cuz you're going so fast. Maybe if you slowed down, China wouldn't be if you slowed down, China wouldn't be if you slowed down, China wouldn't be going so fast either. going so fast either. going so fast either. >> Yeah, at least in part. I think we can >> Yeah, at least in part. I think we can >> Yeah, at least in part. I think we can say the distillation probably plays a say the distillation probably plays a say the distillation probably plays a significant role in cutting into the US significant role in cutting into the US significant role in cutting into the US lead. My sort of mental model for it is, lead. My sort of mental model for it is, lead. My sort of mental model for it is, you know, China is so short on compute you know, China is so short on compute you know, China is so short on compute and that distillation is probably a way and that distillation is probably a way and that distillation is probably a way for them to essentially like train more for them to essentially like train more for them to essentially like train more efficiently, increase the intelligence efficiently, increase the intelligence efficiently, increase the intelligence more efficiently given that they have so more efficiently given that they have so more efficiently given that they have so little compute. So, it's essentially little compute. So, it's essentially little compute. So, it's essentially making up for one of their biggest making up for one of their biggest making up for one of their biggest shortcomings. shortcomings. shortcomings. I don't think that if we suddenly found I don't think that if we suddenly found I don't think that if we suddenly found a way to block all distillation that the a way to block all distillation that the a way to block all distillation that the Chinese labs would just stagnate. You Chinese labs would just stagnate. You Chinese labs would just stagnate. You know, China has, you know, in in AI, in know, China has, you know, in in AI, in know, China has, you know, in in AI, in nuclear weapons, in almost every nuclear weapons, in almost every nuclear weapons, in almost every technical field over the last, you know, technical field over the last, you know, technical field over the last, you know, 30, 40 years, they consistently 30, 40 years, they consistently 30, 40 years, they consistently outperform expectations and just do outperform expectations and just do outperform expectations and just do things that we don't think they should things that we don't think they should things that we don't think they should be able to do given their level of be able to do given their level of be able to do given their level of economic development and their economic development and their economic development and their capabilities. They have an amazing AI capabilities. They have an amazing AI capabilities. They have an amazing AI research ecosystem over there. Like one research ecosystem over there. Like one research ecosystem over there. Like one of the reasons we're ahead is because we of the reasons we're ahead is because we of the reasons we're ahead is because we keep taking Chinese AI researchers and

keep taking Chinese AI researchers and keep taking Chinese AI researchers and employing them in our labs. Like we are, employing them in our labs. Like we are, employing them in our labs. Like we are, you know, siphoning off a lot of their you know, siphoning off a lot of their you know, siphoning off a lot of their top talent. So they have a really top talent. So they have a really top talent. So they have a really thriving ecosystem on its own. But I do thriving ecosystem on its own. But I do thriving ecosystem on its own. But I do think distillation plays a big role. It, think distillation plays a big role. It, think distillation plays a big role. It, you know, it might be the difference you know, it might be the difference you know, it might be the difference between 6 months and a year. It might be between 6 months and a year. It might be between 6 months and a year. It might be the difference between 6 months and 2 the difference between 6 months and 2 the difference between 6 months and 2 years. We don't know. But I think the years. We don't know. But I think the years. We don't know. But I think the fact there's pretty strong evidence that fact there's pretty strong evidence that fact there's pretty strong evidence that the Chinese labs are doing it and they the Chinese labs are doing it and they the Chinese labs are doing it and they wouldn't be doing it if it wasn't to wouldn't be doing it if it wasn't to wouldn't be doing it if it wasn't to their benefit. How does China see us on their benefit. How does China see us on their benefit. How does China see us on AI? How do they see like what our goal AI? How do they see like what our goal AI? How do they see like what our goal actually is? What our goal is v them? So actually is? What our goal is v them? So actually is? What our goal is v them? So as we talk about this question of can as we talk about this question of can as we talk about this question of can these countries cooperate if they need these countries cooperate if they need these countries cooperate if they need to, what is China's perception of to, what is China's perception of to, what is China's perception of America's AI industry? America's AI industry? America's AI industry? >> I think their number one perception is >> I think their number one perception is >> I think their number one perception is that the US wants to hold China down and that the US wants to hold China down and that the US wants to hold China down and wants to constrain China. You know, wants to constrain China. You know, wants to constrain China. You know, especially with the export controls especially with the export controls especially with the export controls >> which came under Biden, I should say. >> which came under Biden, I should say. >> which came under Biden, I should say. >> Yeah. export controls under Biden on >> Yeah. export controls under Biden on >> Yeah. export controls under Biden on these advanced chips. China sees itself these advanced chips. China sees itself these advanced chips. China sees itself as being sort of boxed in by this as being sort of boxed in by this as being sort of boxed in by this hegeimon that wants to kind of keep hegeimon that wants to kind of keep hegeimon that wants to kind of keep China in a permanent position of of China in a permanent position of of China in a permanent position of of subservience. That's a kind of a meta subservience. That's a kind of a meta subservience. That's a kind of a meta narrative across Chinese modern history narrative across Chinese modern history narrative across Chinese modern history and it's one that's crystallized in AI.

and it's one that's crystallized in AI. and it's one that's crystallized in AI. So I think in some ways that's that's So I think in some ways that's that's So I think in some ways that's that's the first thing. Another element is they the first thing. Another element is they the first thing. Another element is they see us often as being pretty see us often as being pretty see us often as being pretty irresponsible, deregulatory, just let it irresponsible, deregulatory, just let it irresponsible, deregulatory, just let it all let it all rip, let it all hang out. all let it all rip, let it all hang out. all let it all rip, let it all hang out. They'll they see sort of chaos within They'll they see sort of chaos within They'll they see sort of chaos within our government. They say oftentimes in our government. They say oftentimes in our government. They say oftentimes in actually in the actually in the actually in the >> that's crazy because it looks so orderly >> that's crazy because it looks so orderly >> that's crazy because it looks so orderly from here [laughter] from here [laughter] from here [laughter] in the in the run-up to uh these in the in the run-up to uh these in the in the run-up to uh these potential AI talks that might be potential AI talks that might be potential AI talks that might be happening in the next couple weeks, happening in the next couple weeks, happening in the next couple weeks, China is issuing sort of opeds by its China is issuing sort of opeds by its China is issuing sort of opeds by its state media where it kind of lays down state media where it kind of lays down state media where it kind of lays down its markers. It tries to position itself its markers. It tries to position itself its markers. It tries to position itself in advance of the talks. And one of the in advance of the talks. And one of the in advance of the talks. And one of the markers that they laid down is, you markers that they laid down is, you markers that they laid down is, you know, America wants to lecture us. They know, America wants to lecture us. They know, America wants to lecture us. They want to tell us what is a safety risk want to tell us what is a safety risk want to tell us what is a safety risk and what isn't. They want to define all and what isn't. They want to define all and what isn't. They want to define all this stuff unilaterally and they don't this stuff unilaterally and they don't this stuff unilaterally and they don't even impose any requirements on their even impose any requirements on their even impose any requirements on their own companies. So don't come to us with own companies. So don't come to us with own companies. So don't come to us with that stuff unless you're going to take that stuff unless you're going to take that stuff unless you're going to take care of your own house. care of your own house. care of your own house. >> Doesn't seem totally unreasonable to me. >> Doesn't seem totally unreasonable to me. >> Doesn't seem totally unreasonable to me. >> Not totally unreasonable. Self-serving >> Not totally unreasonable. Self-serving >> Not totally unreasonable. Self-serving in a way, but uh yes, I mean to in a way, but uh yes, I mean to in a way, but uh yes, I mean to >> but but I think it's interesting. I mean >> but but I think it's interesting. I mean >> but but I think it's interesting. I mean this is a point I was made but to China this is a point I was made but to China this is a point I was made but to China we look like the ones who are not we look like the ones who are not we look like the ones who are not regulating AI that you know there there regulating AI that you know there there regulating AI that you know there there might be this whole discourse and the might be this whole discourse and the might be this whole discourse and the countries need to cooperate but in fact countries need to cooperate but in fact countries need to cooperate but in fact what they see is us racing forward what they see is us racing forward what they see is us racing forward trying to attain AI supremacy before trying to attain AI supremacy before trying to attain AI supremacy before them and kind of in a weird diffuse way them and kind of in a weird diffuse way them and kind of in a weird diffuse way calling for regulation of something that calling for regulation of something that calling for regulation of something that might like destroy all of humanity but might like destroy all of humanity but might like destroy all of humanity but we're not actually doing any serious we're not actually doing any serious we're not actually doing any serious regulation of the thing that might regulation of the thing that might regulation of the thing that might destroy humanity and you know when I

destroy humanity and you know when I destroy humanity and you know when I read some of these state opeds you're read some of these state opeds you're read some of these state opeds you're talking about the way they end up talking about the way they end up talking about the way they end up framing it is insincerity. framing it is insincerity. framing it is insincerity. That That That I never know how when I talk to people I never know how when I talk to people I never know how when I talk to people in Chinese government, their sense of in Chinese government, their sense of in Chinese government, their sense of American politics is actually not often American politics is actually not often American politics is actually not often as sophisticated as I would imagine it as sophisticated as I would imagine it as sophisticated as I would imagine it to be. Maybe I don't get to talk to the to be. Maybe I don't get to talk to the to be. Maybe I don't get to talk to the right people, but I think sometimes they right people, but I think sometimes they right people, but I think sometimes they look at us and assume that the things look at us and assume that the things look at us and assume that the things that are said are have a more orderly that are said are have a more orderly that are said are have a more orderly structure and in the way that everybody structure and in the way that everybody structure and in the way that everybody has to use she's language there. But if has to use she's language there. But if has to use she's language there. But if you look at a thing that doesn't make you look at a thing that doesn't make you look at a thing that doesn't make sense and you come from their sense and you come from their sense and you come from their perspective, well, maybe the reason it perspective, well, maybe the reason it perspective, well, maybe the reason it doesn't make sense is the counterparty doesn't make sense is the counterparty doesn't make sense is the counterparty is not serious. They're just making a is not serious. They're just making a is not serious. They're just making a bunch of different moves that are all bunch of different moves that are all bunch of different moves that are all different forms of a strategy to stay different forms of a strategy to stay different forms of a strategy to stay ahead in the race. as a macro ahead in the race. as a macro ahead in the race. as a macro perception. I mean, I I see this all the perception. I mean, I I see this all the perception. I mean, I I see this all the time talking to Chinese about the US time talking to Chinese about the US time talking to Chinese about the US political system, talking to Americans political system, talking to Americans political system, talking to Americans about the Chinese political system is if about the Chinese political system is if about the Chinese political system is if you don't understand the system at a you don't understand the system at a you don't understand the system at a pretty kind of ground level, if you pretty kind of ground level, if you pretty kind of ground level, if you don't have an intuitive feel for the two don't have an intuitive feel for the two don't have an intuitive feel for the two systems, the tendency is to look at the systems, the tendency is to look at the systems, the tendency is to look at the other side and to connect a bunch of other side and to connect a bunch of other side and to connect a bunch of dots and see a grand conspiracy. And it dots and see a grand conspiracy. And it dots and see a grand conspiracy. And it it usually is a conspiracy against you.

it usually is a conspiracy against you. it usually is a conspiracy against you. And you know the CCP has a very And you know the CCP has a very And you know the CCP has a very conspiratorial view of the world. They conspiratorial view of the world. They conspiratorial view of the world. They see conspiracies everywhere. And you see conspiracies everywhere. And you see conspiracies everywhere. And you know there have been times when the know there have been times when the know there have been times when the United States and other countries have United States and other countries have United States and other countries have conspired to hold down China. But the conspired to hold down China. But the conspired to hold down China. But the way that they will connect dots that way that they will connect dots that way that they will connect dots that from a US perspective are just wildly from a US perspective are just wildly from a US perspective are just wildly unconnected is um you know it's worrying unconnected is um you know it's worrying unconnected is um you know it's worrying and we we basically do the same thing and we we basically do the same thing and we we basically do the same thing over there. We do not have the ability over there. We do not have the ability over there. We do not have the ability to see through rhetoric that's like to see through rhetoric that's like to see through rhetoric that's like that's just what they have to say. that's just what they have to say. that's just what they have to say. That's what they have to say to start That's what they have to say to start That's what they have to say to start and then the real protein the real like and then the real protein the real like and then the real protein the real like meat of this conversation is here. meat of this conversation is here. meat of this conversation is here. That's the real signal. So that's kind That's the real signal. So that's kind That's the real signal. So that's kind of a permanent issue in USChina mutual of a permanent issue in USChina mutual of a permanent issue in USChina mutual perception. perception. perception. >> I don't know from the Chinese >> I don't know from the Chinese >> I don't know from the Chinese perspective and frankly from the perspective and frankly from the perspective and frankly from the American perspective as somebody who American perspective as somebody who American perspective as somebody who does have a good ground level intuition does have a good ground level intuition does have a good ground level intuition for who is saying what in our system and for who is saying what in our system and for who is saying what in our system and why. You know, the Biden export controls why. You know, the Biden export controls why. You know, the Biden export controls on ships were like quite explicitly an on ships were like quite explicitly an on ships were like quite explicitly an effort to maintain AI supremacy for effort to maintain AI supremacy for effort to maintain AI supremacy for America, which is not a crazy thing to America, which is not a crazy thing to America, which is not a crazy thing to do for a country, but if you're the do for a country, but if you're the do for a country, but if you're the country that is being denied the country that is being denied the country that is being denied the exports, I think that's a little bit uh exports, I think that's a little bit uh exports, I think that's a little bit uh provocative. And then Trump comes in, provocative. And then Trump comes in, provocative. And then Trump comes in, right? You know, orients his entire right? You know, orients his entire right? You know, orients his entire trade war against you. you know, Dari trade war against you. you know, Dari trade war against you. you know, Dari Ammedday, the leader of arguably the Ammedday, the leader of arguably the Ammedday, the leader of arguably the most important American AI company in most important American AI company in most important American AI company in anthropic, he had this big essay um on anthropic, he had this big essay um on anthropic, he had this big essay um on the adolescence of technology and he the adolescence of technology and he the adolescence of technology and he says of China, they have hands down the says of China, they have hands down the says of China, they have hands down the clearest path to the AI enabled clearest path to the AI enabled clearest path to the AI enabled totalitarian nightmare I laid out above.

totalitarian nightmare I laid out above. totalitarian nightmare I laid out above. It may even be the default outcome It may even be the default outcome It may even be the default outcome within China as well as within other within China as well as within other within China as well as within other autocratic states to whom the CCP autocratic states to whom the CCP autocratic states to whom the CCP exports surveillance technology. I have exports surveillance technology. I have exports surveillance technology. I have written often about the threat of the written often about the threat of the written often about the threat of the CCP taking the lead in AI and the CCP taking the lead in AI and the CCP taking the lead in AI and the existential imperative to prevent them existential imperative to prevent them existential imperative to prevent them from doing so. So, you know, if I'm from doing so. So, you know, if I'm from doing so. So, you know, if I'm China and I see the leader of, you know, China and I see the leader of, you know, China and I see the leader of, you know, the frontier AI lab saying it's an the frontier AI lab saying it's an the frontier AI lab saying it's an existential imperative to keep China existential imperative to keep China existential imperative to keep China down. I really worry about this down. I really worry about this down. I really worry about this atmosphere atmosphere atmosphere being the one in which these momentous being the one in which these momentous being the one in which these momentous technologies are potentially being technologies are potentially being technologies are potentially being developed because when I talk to developed because when I talk to developed because when I talk to American policy makers, they don't American policy makers, they don't American policy makers, they don't really feel like they understand what is really feel like they understand what is really feel like they understand what is happening in China and there's a lot of happening in China and there's a lot of happening in China and there's a lot of skepticism that an agreement would skepticism that an agreement would skepticism that an agreement would necessarily be verifiable or followed. necessarily be verifiable or followed. necessarily be verifiable or followed. And I think China has pretty good reason And I think China has pretty good reason And I think China has pretty good reason to be skeptical of what our real to be skeptical of what our real to be skeptical of what our real intentions are. our intentions to make intentions are. our intentions to make intentions are. our intentions to make AI safe for everyone or are our AI safe for everyone or are our AI safe for everyone or are our intentions to make sure America is the intentions to make sure America is the intentions to make sure America is the first one with the AI that ensures first one with the AI that ensures first one with the AI that ensures American dominance of the global order American dominance of the global order American dominance of the global order for another 100 or 200 or 500 years. And for another 100 or 200 or 500 years. And for another 100 or 200 or 500 years. And that kind of miasma of mistrust is a that kind of miasma of mistrust is a that kind of miasma of mistrust is a tough space for negotiating. It's a very tough space for negotiating. It's a very tough space for negotiating. It's a very tough space. You know, I mean, this is tough space. You know, I mean, this is tough space. You know, I mean, this is arguably one of the worst times for a arguably one of the worst times for a arguably one of the worst times for a technology this momentous to be coming technology this momentous to be coming technology this momentous to be coming online. You know, at a real moment of online. You know, at a real moment of online. You know, at a real moment of deep geopolitical competition between deep geopolitical competition between deep geopolitical competition between two superpowers that distrust each two superpowers that distrust each two superpowers that distrust each other, that have, you know, interests other, that have, you know, interests other, that have, you know, interests that are fundamentally in conflict in that are fundamentally in conflict in that are fundamentally in conflict in some areas. I think one of the key

some areas. I think one of the key some areas. I think one of the key points here is yes, trust is good. It is points here is yes, trust is good. It is points here is yes, trust is good. It is kind of the lubrication that can ease kind of the lubrication that can ease kind of the lubrication that can ease things along in a negotiation, things along in a negotiation, things along in a negotiation, but it's not going to be the thing that but it's not going to be the thing that but it's not going to be the thing that makes this work or not. Like the thing makes this work or not. Like the thing makes this work or not. Like the thing that makes this work or not, in my that makes this work or not, in my that makes this work or not, in my opinion, is going to be whether or not opinion, is going to be whether or not opinion, is going to be whether or not both sides for their own reasons both sides for their own reasons both sides for their own reasons genuinely believe that this is a genuinely believe that this is a genuinely believe that this is a potentially catastrophic risk and potentially catastrophic risk and potentially catastrophic risk and believe that they need to take action on believe that they need to take action on believe that they need to take action on that for their own safety and security. that for their own safety and security. that for their own safety and security. the Chinese technical AI community needs the Chinese technical AI community needs the Chinese technical AI community needs to like believe deeply in its bones that to like believe deeply in its bones that to like believe deeply in its bones that if we push into this area without the if we push into this area without the if we push into this area without the right safeguards and testing and right safeguards and testing and right safeguards and testing and evaluation then we run a real risk of evaluation then we run a real risk of evaluation then we run a real risk of losing control of this technology and losing control of this technology and losing control of this technology and you know Xiinping other Chinese leaders you know Xiinping other Chinese leaders you know Xiinping other Chinese leaders they do not want that in their own they do not want that in their own they do not want that in their own country for their own reasons and so I country for their own reasons and so I country for their own reasons and so I think that is the area of mutual think that is the area of mutual think that is the area of mutual interest as opposed to mutual trust that interest as opposed to mutual trust that interest as opposed to mutual trust that things have to be built on so when I'm things have to be built on so when I'm things have to be built on so when I'm looking at you know USChina interactions looking at you know USChina interactions looking at you know USChina interactions in this space. I think that one of the in this space. I think that one of the in this space. I think that one of the most important things at least as a most important things at least as a most important things at least as a starting point is can we build up a starting point is can we build up a starting point is can we build up a mutual understanding of the risks? Can mutual understanding of the risks? Can mutual understanding of the risks? Can we share information? What are we seeing we share information? What are we seeing we share information? What are we seeing about emerging risks and how do we test about emerging risks and how do we test about emerging risks and how do we test for those risks? What are the best for those risks? What are the best for those risks? What are the best practices for securing a model that has practices for securing a model that has practices for securing a model that has cyber capabilities that you don't cyber capabilities that you don't cyber capabilities that you don't understand? What are the best practices understand? What are the best practices understand? What are the best practices for sort of defanging a model that might for sort of defanging a model that might for sort of defanging a model that might have bio capabilities that you don't have bio capabilities that you don't have bio capabilities that you don't want? like that work in the United want? like that work in the United want? like that work in the United States is just much more mature. The

States is just much more mature. The States is just much more mature. The regulation in the US is much less regulation in the US is much less regulation in the US is much less mature, but within the companies, within mature, but within the companies, within mature, but within the companies, within the labs, this has been work that the labs, this has been work that the labs, this has been work that they've been doing seriously and they've been doing seriously and they've been doing seriously and investing a lot of money and a lot of investing a lot of money and a lot of investing a lot of money and a lot of people and resources in for a long time. people and resources in for a long time. people and resources in for a long time. And I think that's kind of one of the And I think that's kind of one of the And I think that's kind of one of the misunderstandings in China is they look misunderstandings in China is they look misunderstandings in China is they look entirely at our regulatory ecosystem and entirely at our regulatory ecosystem and entirely at our regulatory ecosystem and they say you're not doing anything. they say you're not doing anything. they say you're not doing anything. Whereas the labs here are voluntarily Whereas the labs here are voluntarily Whereas the labs here are voluntarily doing far more on this than the Chinese doing far more on this than the Chinese doing far more on this than the Chinese labs are doing in sort of a mandatory labs are doing in sort of a mandatory labs are doing in sort of a mandatory regulatory environment. So with all that regulatory environment. So with all that regulatory environment. So with all that work and that knowledge that we've work and that knowledge that we've work and that knowledge that we've gained, can we find ways to safely share gained, can we find ways to safely share gained, can we find ways to safely share some of that with China to essentially some of that with China to essentially some of that with China to essentially seed, bolster, and help grow their seed, bolster, and help grow their seed, bolster, and help grow their existing AI safety ecosystem, their existing AI safety ecosystem, their existing AI safety ecosystem, their technical AI ecosystem over there that technical AI ecosystem over there that technical AI ecosystem over there that is concerned, wants to do good work on is concerned, wants to do good work on is concerned, wants to do good work on this, but is just starting, you know, this, but is just starting, you know, this, but is just starting, you know, five plus years later and just has five plus years later and just has five plus years later and just has invested far less people, money, invested far less people, money, invested far less people, money, computing resources in that work. So you computing resources in that work. So you computing resources in that work. So you have some personal experience here. have some personal experience here. have some personal experience here. You've hosted some of these China US AI You've hosted some of these China US AI You've hosted some of these China US AI dialogues. Not the official highle US dialogues. Not the official highle US dialogues. Not the official highle US government ones, but these more informal government ones, but these more informal government ones, but these more informal ones that are, I think, in some ways ones that are, I think, in some ways ones that are, I think, in some ways supposed to help lay long-term supposed to help lay long-term supposed to help lay long-term groundwork for this. What have those groundwork for this. What have those groundwork for this. What have those felt like? What have you learned from felt like? What have you learned from felt like? What have you learned from interacting with Chinese colleagues and interacting with Chinese colleagues and interacting with Chinese colleagues and counterparts? Like give me some of your counterparts? Like give me some of your counterparts? Like give me some of your texture on this. You know, these texture on this. You know, these texture on this. You know, these conversations are normally very very conversations are normally very very conversations are normally very very technocratic. you know, maybe a little technocratic. you know, maybe a little technocratic. you know, maybe a little bit boring, productive, but you know, bit boring, productive, but you know, bit boring, productive, but you know, calm affairs. And I think one thing I've calm affairs. And I think one thing I've calm affairs. And I think one thing I've seen a couple times when you get a seen a couple times when you get a seen a couple times when you get a little bit of heat, like a little bit of little bit of heat, like a little bit of little bit of heat, like a little bit of spark in the conversation happens often

spark in the conversation happens often spark in the conversation happens often times when the Americans are pointing times when the Americans are pointing times when the Americans are pointing at, you know, these trend lines and AI at, you know, these trend lines and AI at, you know, these trend lines and AI and bio or these uh emerging safety and bio or these uh emerging safety and bio or these uh emerging safety issues scaling and they say like look at issues scaling and they say like look at issues scaling and they say like look at this like why aren't you more concerned this like why aren't you more concerned this like why aren't you more concerned about this? Why aren't you doing more on about this? Why aren't you doing more on about this? Why aren't you doing more on safety? And you'll see the training side safety? And you'll see the training side safety? And you'll see the training side getting actually really frustrated and getting actually really frustrated and getting actually really frustrated and being like you guys don't get it. We we being like you guys don't get it. We we being like you guys don't get it. We we are doing a ton on safety. We have these are doing a ton on safety. We have these are doing a ton on safety. We have these AI companion regulations. We have uh AI companion regulations. We have uh AI companion regulations. We have uh mandatory labeling of AI generated mandatory labeling of AI generated mandatory labeling of AI generated content. We have all of these rules and content. We have all of these rules and content. We have all of these rules and regulations. They're just not the exact regulations. They're just not the exact regulations. They're just not the exact thing that you want. And I think that's thing that you want. And I think that's thing that you want. And I think that's a lot of the disconnect between the two a lot of the disconnect between the two a lot of the disconnect between the two sides is the Chinese side feels like sides is the Chinese side feels like sides is the Chinese side feels like they have been doing serious work for a they have been doing serious work for a they have been doing serious work for a long time and that's just not seen. It's long time and that's just not seen. It's long time and that's just not seen. It's not [music] understood or not respected not [music] understood or not respected not [music] understood or not respected outside of the country. outside of the country. outside of the country. >> If you're not a subscriber to the New >> If you're not a subscriber to the New >> If you're not a subscriber to the New York Times, we [music] have some news York Times, we [music] have some news York Times, we [music] have some news for you. You can now explore the Times for you. You can now explore the Times for you. You can now explore the Times for free without any payw walls at all for free without any payw walls at all for free without any payw walls at all during your first month in the New York during your first month in the New York during your first month in the New York [music] Times app.

I think talking about this requires some I think talking about this requires some sense of how China's AI industry differs sense of how China's AI industry differs sense of how China's AI industry differs from ours. from ours. from ours. How would you describe the difference How would you describe the difference How would you describe the difference between what the sort of culture and between what the sort of culture and between what the sort of culture and approach of the frontier AI makers in approach of the frontier AI makers in approach of the frontier AI makers in China is compared to sort of the China is compared to sort of the China is compared to sort of the anthropic open AI you know Google deep anthropic open AI you know Google deep anthropic open AI you know Google deep mind here so I think among the most mind here so I think among the most mind here so I think among the most frontier labs I'd say that's where the frontier labs I'd say that's where the frontier labs I'd say that's where the cultural similarities are the closest cultural similarities are the closest cultural similarities are the closest and it's much more of the broader AI and it's much more of the broader AI and it's much more of the broader AI industry and the policy ecosystem industry and the policy ecosystem industry and the policy ecosystem where the differences are much wider and where the differences are much wider and where the differences are much wider and I'll start with that sort of broader I'll start with that sort of broader I'll start with that sort of broader ecosystem and then kind of bring it into ecosystem and then kind of bring it into ecosystem and then kind of bring it into the frontier. I think in many ways in the frontier. I think in many ways in the frontier. I think in many ways in the US a huge portion of the AI industry the US a huge portion of the AI industry the US a huge portion of the AI industry and the policy world really started from and the policy world really started from and the policy world really started from this idea of one day we will reach super this idea of one day we will reach super this idea of one day we will reach super intelligence that is the goal that will intelligence that is the goal that will intelligence that is the goal that will bring with it catastrophic risks that bring with it catastrophic risks that bring with it catastrophic risks that will require heavy focus on safety and will require heavy focus on safety and will require heavy focus on safety and it's been this this magnet that's like it's been this this magnet that's like it's been this this magnet that's like drawing us into this future and that's drawing us into this future and that's drawing us into this future and that's of course especially true at anthropic of course especially true at anthropic of course especially true at anthropic and at openai and deep mind but I think and at openai and deep mind but I think and at openai and deep mind but I think that that's a pretty significant portion that that's a pretty significant portion that that's a pretty significant portion of the policy eosystem system here too.

of the policy eosystem system here too. of the policy eosystem system here too. And so it's almost like it's like this And so it's almost like it's like this And so it's almost like it's like this teological thing of we're endlessly teological thing of we're endlessly teological thing of we're endlessly being drawn towards that. And in China being drawn towards that. And in China being drawn towards that. And in China there are some people a couple of people there are some people a couple of people there are some people a couple of people who lead the frontier labs who have that who lead the frontier labs who have that who lead the frontier labs who have that take but in terms of a broad culture take but in terms of a broad culture take but in terms of a broad culture throughout the industry the investors throughout the industry the investors throughout the industry the investors the engineers the policy people the the engineers the policy people the the engineers the policy people the government that has not been this kind government that has not been this kind government that has not been this kind of magnet drawing them into the future of magnet drawing them into the future of magnet drawing them into the future in the same way. It's more like they've in the same way. It's more like they've in the same way. It's more like they've been developing an industry, developing been developing an industry, developing been developing an industry, developing applications of it. As they develop a applications of it. As they develop a applications of it. As they develop a new application, they develop new policy new application, they develop new policy new application, they develop new policy to deal with that. It's a pretty to deal with that. It's a pretty to deal with that. It's a pretty fundamental difference in the way the fundamental difference in the way the fundamental difference in the way the ecosystems have kind of grown and ecosystems have kind of grown and ecosystems have kind of grown and expanded. I've thought about this a lot expanded. I've thought about this a lot expanded. I've thought about this a lot in the American context. Something I in the American context. Something I in the American context. Something I sometimes say to American policy makers sometimes say to American policy makers sometimes say to American policy makers who are like, where do you start? who are like, where do you start? who are like, where do you start? Someone say, well, you start by starting Someone say, well, you start by starting Someone say, well, you start by starting that you learn how to regulate things. that you learn how to regulate things. that you learn how to regulate things. You learn how to legislate on them by You learn how to legislate on them by You learn how to legislate on them by regulating and legislating on them. and regulating and legislating on them. and regulating and legislating on them. and the Chinese approach that they already the Chinese approach that they already the Chinese approach that they already are learning day by day how to interface are learning day by day how to interface are learning day by day how to interface with their labs, how to craft with their labs, how to craft with their labs, how to craft regulations and revise them that they regulations and revise them that they regulations and revise them that they are building up practical regulatory are building up practical regulatory are building up practical regulatory experience experience experience that then that then that then if or when they need to come in with if or when they need to come in with if or when they need to come in with things that are much more potent, they things that are much more potent, they things that are much more potent, they sort of know how to do that. And so it's sort of know how to do that. And so it's sort of know how to do that. And so it's not that their regulations are what the not that their regulations are what the not that their regulations are what the American Frontier Labs believe are American Frontier Labs believe are American Frontier Labs believe are needed or what I believe are needed.

needed or what I believe are needed. needed or what I believe are needed. They're not. Although frankly, I would They're not. Although frankly, I would They're not. Although frankly, I would like us to be more thoughtful about AI like us to be more thoughtful about AI like us to be more thoughtful about AI companion bots than we've been. That companion bots than we've been. That companion bots than we've been. That we're actually behind. And practical we're actually behind. And practical we're actually behind. And practical experience here is meaningful. In some experience here is meaningful. In some experience here is meaningful. In some ways, it's more meaningful than like ways, it's more meaningful than like ways, it's more meaningful than like neat conceptual arguments about, you neat conceptual arguments about, you neat conceptual arguments about, you know, the worst case outcomes. You know, know, the worst case outcomes. You know, know, the worst case outcomes. You know, one of the characteristics of Chinese one of the characteristics of Chinese one of the characteristics of Chinese policym, but especially in AI, is that policym, but especially in AI, is that policym, but especially in AI, is that it it's very iterative. They'll roll out it it's very iterative. They'll roll out it it's very iterative. They'll roll out a regulation. They'll see how it's a regulation. They'll see how it's a regulation. They'll see how it's working. They'll roll out a technical working. They'll roll out a technical working. They'll roll out a technical standard that specifies it. It's not standard that specifies it. It's not standard that specifies it. It's not quite achieving the end that they want, quite achieving the end that they want, quite achieving the end that they want, and they'll roll out another regulation and they'll roll out another regulation and they'll roll out another regulation that basically just overlaps on the that basically just overlaps on the that basically just overlaps on the first one. And with each one of these, first one. And with each one of these, first one. And with each one of these, they've built up these reusable they've built up these reusable they've built up these reusable regulatory tools. So, the main one is regulatory tools. So, the main one is regulatory tools. So, the main one is this uh registration system for AI this uh registration system for AI this uh registration system for AI models. and that the CAC, the Cyerspace models. and that the CAC, the Cyerspace models. and that the CAC, the Cyerspace Administration of China, the main Administration of China, the main Administration of China, the main regulator there, needs to be able to regulator there, needs to be able to regulator there, needs to be able to read, needs to be able to understand, in read, needs to be able to understand, in read, needs to be able to understand, in some cases, maybe do the tests on their some cases, maybe do the tests on their some cases, maybe do the tests on their own. And you know, when they first own. And you know, when they first own. And you know, when they first started this in 2021, started this in 2021, started this in 2021, they were the regulators were were they were the regulators were were they were the regulators were were totally out of their depth. Like the CAC totally out of their depth. Like the CAC totally out of their depth. Like the CAC is traditionally an internet regulator.

is traditionally an internet regulator. is traditionally an internet regulator. It's focused on, you know, content and It's focused on, you know, content and It's focused on, you know, content and political content, stuff like that. But political content, stuff like that. But political content, stuff like that. But that was 4 years ago. And you know that that was 4 years ago. And you know that that was 4 years ago. And you know that like I said has been focused initially like I said has been focused initially like I said has been focused initially all on this controlling content. It's all on this controlling content. It's all on this controlling content. It's now been expanded these other areas now been expanded these other areas now been expanded these other areas about you know emotional dependency about you know emotional dependency about you know emotional dependency around sort of labeling of content. You around sort of labeling of content. You around sort of labeling of content. You know can you impose and then remove a know can you impose and then remove a know can you impose and then remove a label on AI generated content stuff like label on AI generated content stuff like label on AI generated content stuff like that but they have been sort of that but they have been sort of that but they have been sort of constantly in touch with the labs for constantly in touch with the labs for constantly in touch with the labs for now about four years. So that's that's a now about four years. So that's that's a now about four years. So that's that's a lot of regulatory practice and lot of regulatory practice and lot of regulatory practice and regulatory muscle. regulatory muscle. regulatory muscle. Part of the question is with these Part of the question is with these Part of the question is with these frontier safety risks, is that something frontier safety risks, is that something frontier safety risks, is that something that you can kind of just easily tack on that you can kind of just easily tack on that you can kind of just easily tack on to this? Is it just another test that to this? Is it just another test that to this? Is it just another test that they run or is it something they run or is it something they run or is it something significantly more complicated? And I significantly more complicated? And I significantly more complicated? And I think it, you know, it's kind of in think it, you know, it's kind of in think it, you know, it's kind of in between the two. Like they have a lot of between the two. Like they have a lot of between the two. Like they have a lot of mechanisms, they have a lot of habits mechanisms, they have a lot of habits mechanisms, they have a lot of habits and and touch points that are very good, and and touch points that are very good, and and touch points that are very good, but they do need to increase their but they do need to increase their but they do need to increase their technical capabilities in these specific technical capabilities in these specific technical capabilities in these specific areas of frontier risk and control. Tell areas of frontier risk and control. Tell areas of frontier risk and control. Tell me about the way in which China's me about the way in which China's me about the way in which China's evolved to emphasize openw weight models evolved to emphasize openw weight models evolved to emphasize openw weight models versus our uh main models anthropic open versus our uh main models anthropic open versus our uh main models anthropic open AI are closed weight and maybe begin for AI are closed weight and maybe begin for AI are closed weight and maybe begin for people who don't know those terms by people who don't know those terms by people who don't know those terms by defining them.

defining them. defining them. >> Sure. So closed models are the way that >> Sure. So closed models are the way that >> Sure. So closed models are the way that when you use chat GPT or claude or when you use chat GPT or claude or when you use chat GPT or claude or Gemini those are closed models as in you Gemini those are closed models as in you Gemini those are closed models as in you interact with it kind of on the interact with it kind of on the interact with it kind of on the company's terms through their portal. company's terms through their portal. company's terms through their portal. You cannot sort of edit the model. You You cannot sort of edit the model. You You cannot sort of edit the model. You cannot download it to your computer and cannot download it to your computer and cannot download it to your computer and run it yourself. An openweight model can run it yourself. An openweight model can run it yourself. An openweight model can be downloaded from the internet. And if be downloaded from the internet. And if be downloaded from the internet. And if you know how to do it, you can play with you know how to do it, you can play with you know how to do it, you can play with it. You can tweak it. You can remove it. You can tweak it. You can remove it. You can tweak it. You can remove safeguards. You can add new safeguards. You can add new safeguards. You can add new capabilities. You can sort of tailor it capabilities. You can sort of tailor it capabilities. You can sort of tailor it to your own purposes. If you need to use to your own purposes. If you need to use to your own purposes. If you need to use a model and you need to make, you know, a model and you need to make, you know, a model and you need to make, you know, thousands or tens of thousands of calls thousands or tens of thousands of calls thousands or tens of thousands of calls of it every single day to run your own of it every single day to run your own of it every single day to run your own startup, you do not want to be paying startup, you do not want to be paying startup, you do not want to be paying anthropic and open AI for every single anthropic and open AI for every single anthropic and open AI for every single one of those tokens every time you ask one of those tokens every time you ask one of those tokens every time you ask the model a question. And this has been the model a question. And this has been the model a question. And this has been a divide that's really emerged starting a divide that's really emerged starting a divide that's really emerged starting especially in like 2024 or so where it especially in like 2024 or so where it especially in like 2024 or so where it wasn't always a given that this is how wasn't always a given that this is how wasn't always a given that this is how the two ecosystems would develop. But the two ecosystems would develop. But the two ecosystems would develop. But the way it has developed is that Chinese the way it has developed is that Chinese the way it has developed is that Chinese labs primarily release their models open labs primarily release their models open labs primarily release their models open weight and the US labs primarily release weight and the US labs primarily release weight and the US labs primarily release them closed weight. So at the at the them closed weight. So at the at the them closed weight. So at the at the very beginning sort of in the aftermath very beginning sort of in the aftermath very beginning sort of in the aftermath of chat GPT when China was first of chat GPT when China was first of chat GPT when China was first regulating generative AI they actually regulating generative AI they actually regulating generative AI they actually started off taking a relatively cautious started off taking a relatively cautious started off taking a relatively cautious approach to openweight models and approach to openweight models and approach to openweight models and putting regulatory burdens on them that putting regulatory burdens on them that putting regulatory burdens on them that would have made it much harder to use would have made it much harder to use would have made it much harder to use openweight models in China. And the openweight models in China. And the openweight models in China. And the reason they were doing that is because reason they were doing that is because reason they were doing that is because at the time the primary the leading at the time the primary the leading at the time the primary the leading openweight model was Llama from Meta openweight model was Llama from Meta openweight model was Llama from Meta from Facebook and China was worried you

from Facebook and China was worried you from Facebook and China was worried you know our Chinese developers going to know our Chinese developers going to know our Chinese developers going to take in Llama they're going to build take in Llama they're going to build take in Llama they're going to build their applications on top of it and it's their applications on top of it and it's their applications on top of it and it's going to kind of poison our ecosystem going to kind of poison our ecosystem going to kind of poison our ecosystem with their information that we don't with their information that we don't with their information that we don't want. But over the next year or so, we want. But over the next year or so, we want. But over the next year or so, we saw a couple of the leading Chinese labs saw a couple of the leading Chinese labs saw a couple of the leading Chinese labs decide to release their models decide to release their models decide to release their models openweight. And you know, Deepseek was openweight. And you know, Deepseek was openweight. And you know, Deepseek was really the big kind of kaboom moment in really the big kind of kaboom moment in really the big kind of kaboom moment in this in that when they released it open this in that when they released it open this in that when they released it open weight, it took the world by storm. You weight, it took the world by storm. You weight, it took the world by storm. You know, the entire global AI community was know, the entire global AI community was know, the entire global AI community was able to actually play with it and look able to actually play with it and look able to actually play with it and look at it and see that it really is that at it and see that it really is that at it and see that it really is that impressive. And since then, it's kind of impressive. And since then, it's kind of impressive. And since then, it's kind of snowballed from there. And I think in snowballed from there. And I think in snowballed from there. And I think in some ways the CCP might have stumbled some ways the CCP might have stumbled some ways the CCP might have stumbled into this outcome, but I think they're into this outcome, but I think they're into this outcome, but I think they're pretty happy with it. And it makes sense pretty happy with it. And it makes sense pretty happy with it. And it makes sense both for the companies to a certain both for the companies to a certain both for the companies to a certain extent and for the government. Doesn't extent and for the government. Doesn't extent and for the government. Doesn't it make it harder to control these it make it harder to control these it make it harder to control these models? One thing going on in and I mean models? One thing going on in and I mean models? One thing going on in and I mean this is a debate in the American AI this is a debate in the American AI this is a debate in the American AI ecosystem where you know Dario Amade and ecosystem where you know Dario Amade and ecosystem where you know Dario Amade and Sam Alman and like they often fight with Sam Alman and like they often fight with Sam Alman and like they often fight with Mark Zuckerberg over this. There's a Mark Zuckerberg over this. There's a Mark Zuckerberg over this. There's a view that when you get these very view that when you get these very view that when you get these very powerful models like something like, you powerful models like something like, you powerful models like something like, you know, Mythos, which has these incredible know, Mythos, which has these incredible know, Mythos, which has these incredible cyber hacking implications, cyber hacking implications, cyber hacking implications, you don't want anybody to be able to you don't want anybody to be able to you don't want anybody to be able to just download Mythos and do what they just download Mythos and do what they just download Mythos and do what they want with it. I mean, these are potent want with it. I mean, these are potent want with it. I mean, these are potent things. You need to have some control things. You need to have some control things. You need to have some control over them. The CCP is both has a more over them. The CCP is both has a more over them. The CCP is both has a more aggressive regulatory stance and is more aggressive regulatory stance and is more aggressive regulatory stance and is more control obsessed than the US government control obsessed than the US government control obsessed than the US government tends to be. And yet China is the center

tends to be. And yet China is the center tends to be. And yet China is the center of the openweight ecosystem. Like how of the openweight ecosystem. Like how of the openweight ecosystem. Like how did that how do those things hold did that how do those things hold did that how do those things hold together? I think one factor is like together? I think one factor is like together? I think one factor is like what was needed for the companies to be what was needed for the companies to be what was needed for the companies to be seen as globally competitive. You know I seen as globally competitive. You know I seen as globally competitive. You know I think if it deepseek in late 2024 had think if it deepseek in late 2024 had think if it deepseek in late 2024 had just announced to the world hey we've just announced to the world hey we've just announced to the world hey we've got a great model and feel free to use got a great model and feel free to use got a great model and feel free to use it. It'll go to Chinese servers and it. It'll go to Chinese servers and it. It'll go to Chinese servers and we'll give you back the answers. I think we'll give you back the answers. I think we'll give you back the answers. I think there would have been a level of there would have been a level of there would have been a level of suspicion about that. I don't think it suspicion about that. I don't think it suspicion about that. I don't think it would have seen this rapid global would have seen this rapid global would have seen this rapid global proliferation because people have a proliferation because people have a proliferation because people have a certain distrust of Chinese technology certain distrust of Chinese technology certain distrust of Chinese technology and by releasing it open weight they can and by releasing it open weight they can and by releasing it open weight they can essentially say hey you look at it you essentially say hey you look at it you essentially say hey you look at it you change it you do whatever you want to it change it you do whatever you want to it change it you do whatever you want to it it's that good and you will see that and it's that good and you will see that and it's that good and you will see that and then we'll figure out how to make money then we'll figure out how to make money then we'll figure out how to make money other ways so I think that's part of the other ways so I think that's part of the other ways so I think that's part of the business uh aspect to this it's hugely business uh aspect to this it's hugely business uh aspect to this it's hugely reputation enhancing for Chinese reputation enhancing for Chinese reputation enhancing for Chinese companies and now for China's AI companies and now for China's AI companies and now for China's AI ecosystem as a whole to release these ecosystem as a whole to release these ecosystem as a whole to release these open weight and therefore or overcome open weight and therefore or overcome open weight and therefore or overcome some of the suspicion that normally some of the suspicion that normally some of the suspicion that normally falls on Chinese companies when they go falls on Chinese companies when they go falls on Chinese companies when they go global. So that's one part of it.

global. So that's one part of it. global. So that's one part of it. Another part is that you know frankly Another part is that you know frankly Another part is that you know frankly these are probably undermining the these are probably undermining the these are probably undermining the future valuation of anthropic and open future valuation of anthropic and open future valuation of anthropic and open AI and I don't think that that was a AI and I don't think that that was a AI and I don't think that that was a scheme going back to 2023 or 2024 when scheme going back to 2023 or 2024 when scheme going back to 2023 or 2024 when you know this world we're in wasn't you know this world we're in wasn't you know this world we're in wasn't totally foreseeable. Now that they're totally foreseeable. Now that they're totally foreseeable. Now that they're here, I think it says well that you know here, I think it says well that you know here, I think it says well that you know that that is to our benefit in terms of that that is to our benefit in terms of that that is to our benefit in terms of long-term competitiveness. long-term competitiveness. long-term competitiveness. >> Do the Chinese models give extremely >> Do the Chinese models give extremely >> Do the Chinese models give extremely different answers or come up with very different answers or come up with very different answers or come up with very different approaches in the American different approaches in the American different approaches in the American models? Is a more fundamentally models? Is a more fundamentally models? Is a more fundamentally different worldview different worldview different worldview detectable if you kind of run testing detectable if you kind of run testing detectable if you kind of run testing across the two, right? Uh more across the two, right? Uh more across the two, right? Uh more skepticism of democracy generally. You skepticism of democracy generally. You skepticism of democracy generally. You know, I think I think American models know, I think I think American models know, I think I think American models very much do have an American outlook on very much do have an American outlook on very much do have an American outlook on the world. Do you see the models as the world. Do you see the models as the world. Do you see the models as being very different when you know when being very different when you know when being very different when you know when Americans are talking to Deepseek? It Americans are talking to Deepseek? It Americans are talking to Deepseek? It depends a little bit on how you're using depends a little bit on how you're using depends a little bit on how you're using the model. Like the most censored the model. Like the most censored the model. Like the most censored version of a Chinese model will be when version of a Chinese model will be when version of a Chinese model will be when you're using it through the app or you're using it through the app or you're using it through the app or you're using it through the API when you're using it through the API when you're using it through the API when you're going to deepsee.com and asking you're going to deepsee.com and asking you're going to deepsee.com and asking it questions. That's the version that it questions. That's the version that it questions. That's the version that has sort of the most controls built into has sort of the most controls built into has sort of the most controls built into it. If you download the model um an open it. If you download the model um an open it. If you download the model um an open weight model, you download it, you run weight model, you download it, you run weight model, you download it, you run it on your own computer, you will have a it on your own computer, you will have a it on your own computer, you will have a different set of safeguards, not different set of safeguards, not different set of safeguards, not entirely removed, but a different set of entirely removed, but a different set of entirely removed, but a different set of them. And you can also tweak those, you them. And you can also tweak those, you them. And you can also tweak those, you can remove some of those, you can add can remove some of those, you can add can remove some of those, you can add new training data, you can you can new training data, you can you can new training data, you can you can change the way that the model functions.

change the way that the model functions. change the way that the model functions. And some American companies like Curser And some American companies like Curser And some American companies like Curser and others, they feel that they can get and others, they feel that they can get and others, they feel that they can get the models into a place where they are the models into a place where they are the models into a place where they are not, you know, propaganda machines for not, you know, propaganda machines for not, you know, propaganda machines for uh for the CCP. And I think the CCP uh for the CCP. And I think the CCP uh for the CCP. And I think the CCP would sort of say that you know there would sort of say that you know there would sort of say that you know there are a lot of countries Singapore, are a lot of countries Singapore, are a lot of countries Singapore, Southeast Asian countries that are Southeast Asian countries that are Southeast Asian countries that are building sovereign AI models on top of building sovereign AI models on top of building sovereign AI models on top of these openweight models. Say you know these openweight models. Say you know these openweight models. Say you know add your own language data, add your own add your own language data, add your own add your own language data, add your own cultural data to sort of post-train cultural data to sort of post-train cultural data to sort of post-train these models in a way and tweak them to these models in a way and tweak them to these models in a way and tweak them to your needs. So that's that's part of your needs. So that's that's part of your needs. So that's that's part of China's pitch to the world to the global China's pitch to the world to the global China's pitch to the world to the global south is America is the technological south is America is the technological south is America is the technological hgeimon that wants to restrict your hgeimon that wants to restrict your hgeimon that wants to restrict your access to this technology. It wants to access to this technology. It wants to access to this technology. It wants to impose its own values. It wants to, you impose its own values. It wants to, you impose its own values. It wants to, you know, blot out your own local culture know, blot out your own local culture know, blot out your own local culture and it won't let you, it won't let you and it won't let you, it won't let you and it won't let you, it won't let you in any way adapt or play with their in any way adapt or play with their in any way adapt or play with their models. We're just giving you the model models. We're just giving you the model models. We're just giving you the model and you can do with it what you want. and you can do with it what you want. and you can do with it what you want. You can change it. You can adapt it and You can change it. You can adapt it and You can change it. You can adapt it and you can run it for just the cost of, you you can run it for just the cost of, you you can run it for just the cost of, you know, the cloud computing that you're know, the cloud computing that you're know, the cloud computing that you're using. That's the pitch. I don't think using. That's the pitch. I don't think using. That's the pitch. I don't think it's I don't think it's 100% honest. I it's I don't think it's 100% honest. I it's I don't think it's 100% honest. I don't think it's actually going to play don't think it's actually going to play don't think it's actually going to play out in that exact way, but that's the out in that exact way, but that's the out in that exact way, but that's the the divide that China has been trying to the divide that China has been trying to the divide that China has been trying to pitch to the rest of the world. So, so I pitch to the rest of the world. So, so I pitch to the rest of the world. So, so I find this really interesting. So, the find this really interesting. So, the find this really interesting. So, the way the internet developed and a lot of way the internet developed and a lot of way the internet developed and a lot of the modern mega online platforms the modern mega online platforms the modern mega online platforms developed, China and the US have pretty developed, China and the US have pretty developed, China and the US have pretty separated digital ecosystems. I mean, separated digital ecosystems. I mean, separated digital ecosystems. I mean, you're not using a lot of Google search you're not using a lot of Google search you're not using a lot of Google search in China. We're not using WeChat here.

in China. We're not using WeChat here. in China. We're not using WeChat here. We are much more integrated on AI than We are much more integrated on AI than We are much more integrated on AI than we are on what came before. A pretty we are on what came before. A pretty we are on what came before. A pretty large number of American companies which large number of American companies which large number of American companies which are consuming AI tokens at a level where are consuming AI tokens at a level where are consuming AI tokens at a level where you actually have to pay real money to you actually have to pay real money to you actually have to pay real money to keep going. They're using Chinese keep going. They're using Chinese keep going. They're using Chinese models. I mean Airbnb, Coinbase are models. I mean Airbnb, Coinbase are models. I mean Airbnb, Coinbase are famously using Chinese models for famously using Chinese models for famously using Chinese models for significant parts of their AI significant parts of their AI significant parts of their AI infrastructure. And so this is not just infrastructure. And so this is not just infrastructure. And so this is not just like the Chinese ecosystem over here and like the Chinese ecosystem over here and like the Chinese ecosystem over here and the American ecosystem over here. the American ecosystem over here. the American ecosystem over here. They're already somewhat combined. I They're already somewhat combined. I They're already somewhat combined. I don't know how much open AI or cloud are don't know how much open AI or cloud are don't know how much open AI or cloud are allowed in China because I assume allowed in China because I assume allowed in China because I assume they're not censoring in the way that they're not censoring in the way that they're not censoring in the way that the CCP would want them to. But the the CCP would want them to. But the the CCP would want them to. But the Chinese models are are here and in Chinese models are are here and in Chinese models are are here and in widespread widespread widespread commercial use. commercial use. commercial use. >> Yeah, this is really one of like the >> Yeah, this is really one of like the >> Yeah, this is really one of like the great ironies of this current AI moment great ironies of this current AI moment great ironies of this current AI moment that we're in. that we're in. that we're in. The the firewall really came down and The the firewall really came down and The the firewall really came down and kicked out the American technology kicked out the American technology kicked out the American technology companies, Google, Facebook, Twitter, companies, Google, Facebook, Twitter, companies, Google, Facebook, Twitter, etc. 2008 2010. You know, we had very etc. 2008 2010. You know, we had very etc. 2008 2010. You know, we had very separated product ecosystems. They were separated product ecosystems. They were separated product ecosystems. They were building their own products. We were building their own products. We were building their own products. We were building our own products. The products building our own products. The products building our own products. The products didn't really cross over, but we always didn't really cross over, but we always didn't really cross over, but we always actually had pretty integrated sort of actually had pretty integrated sort of actually had pretty integrated sort of technology ecosystems. You had a huge technology ecosystems. You had a huge technology ecosystems. You had a huge flow of Chinese people coming to the US flow of Chinese people coming to the US flow of Chinese people coming to the US and working in companies. A lot of them and working in companies. A lot of them and working in companies. A lot of them would go back and kind of would go back and kind of would go back and kind of cross-pollinate the two ecosystems with cross-pollinate the two ecosystems with cross-pollinate the two ecosystems with ideas. We had a lot of American money ideas. We had a lot of American money ideas. We had a lot of American money going into Chinese startups, a lot of going into Chinese startups, a lot of going into Chinese startups, a lot of Chinese money going into American Chinese money going into American Chinese money going into American startups. It was all quite integrated startups. It was all quite integrated startups. It was all quite integrated outside of the product layer up till outside of the product layer up till outside of the product layer up till about 2017 2018.

about 2017 2018. about 2017 2018. That's when you know we began the the That's when you know we began the the That's when you know we began the the American project of technology American project of technology American project of technology decoupling with China. We want to pull decoupling with China. We want to pull decoupling with China. We want to pull apart these connections because we think apart these connections because we think apart these connections because we think this is how China is catching up. It's this is how China is catching up. It's this is how China is catching up. It's catching up because they're stealing. catching up because they're stealing. catching up because they're stealing. It's catching up because they're It's catching up because they're It's catching up because they're learning at our universities etc etc. learning at our universities etc etc. learning at our universities etc etc. And so you know the first Trump And so you know the first Trump And so you know the first Trump administration to a certain extent the administration to a certain extent the administration to a certain extent the Biden administration did a lot to cut Biden administration did a lot to cut Biden administration did a lot to cut down the flows of people to cut off the down the flows of people to cut off the down the flows of people to cut off the flows of money to kind of reduce the flows of money to kind of reduce the flows of money to kind of reduce the flow of ideas between the two ecosystems flow of ideas between the two ecosystems flow of ideas between the two ecosystems and that was relatively successful and I and that was relatively successful and I and that was relatively successful and I think it probably would have continued think it probably would have continued think it probably would have continued to be quite segmented in this way except to be quite segmented in this way except to be quite segmented in this way except for the fact that the Chinese model is for the fact that the Chinese model is for the fact that the Chinese model is going open weight and so it's almost going open weight and so it's almost going open weight and so it's almost like the open weight ecosystem is kind like the open weight ecosystem is kind like the open weight ecosystem is kind of reintegrating these ecosystems in a of reintegrating these ecosystems in a of reintegrating these ecosystems in a way that I don't think anybody could way that I don't think anybody could way that I don't think anybody could have foreseen 3 have foreseen 3 have foreseen 3 10 years ago. 10 years ago. 10 years ago. >> Xi Jinping recently gave a pretty big >> Xi Jinping recently gave a pretty big >> Xi Jinping recently gave a pretty big speech on AI. What seemed new to you in speech on AI. What seemed new to you in speech on AI. What seemed new to you in that speech? that speech? that speech? >> So this speech was at the World AI >> So this speech was at the World AI >> So this speech was at the World AI conference which is China's premier uh conference which is China's premier uh conference which is China's premier uh AI event every year. They try to get you AI event every year. They try to get you AI event every year. They try to get you know the whole world to come out. It's a know the whole world to come out. It's a know the whole world to come out. It's a big to-do and this is the first year big to-do and this is the first year big to-do and this is the first year that Xiinping has attended and given a that Xiinping has attended and given a that Xiinping has attended and given a speech there. So, it's really his speech there. So, it's really his speech there. So, it's really his biggest AI speech maybe ever. So, people biggest AI speech maybe ever. So, people biggest AI speech maybe ever. So, people were watching it very closely. I think a were watching it very closely. I think a were watching it very closely. I think a good portion of it was China's pitch to good portion of it was China's pitch to good portion of it was China's pitch to the rest of the world. It's the one I the rest of the world. It's the one I the rest of the world. It's the one I outlined earlier. And then the other outlined earlier. And then the other outlined earlier. And then the other part that stuck out to me was the part that stuck out to me was the part that stuck out to me was the conclusion. And it ended with some conclusion. And it ended with some conclusion. And it ended with some pretty striking uh metaphors using an pretty striking uh metaphors using an pretty striking uh metaphors using an ancient Chinese idiom that I don't have

ancient Chinese idiom that I don't have ancient Chinese idiom that I don't have off the top of my head about how sort of off the top of my head about how sort of off the top of my head about how sort of the the wise adapt to circumstances and the the wise adapt to circumstances and the the wise adapt to circumstances and they do not get stuck on one path. And I they do not get stuck on one path. And I they do not get stuck on one path. And I think the one of the key terms was that think the one of the key terms was that think the one of the key terms was that we need to be able to act to forestall we need to be able to act to forestall we need to be able to act to forestall loss of control of AI. And this has been loss of control of AI. And this has been loss of control of AI. And this has been a is a long-term concern in the west. a is a long-term concern in the west. a is a long-term concern in the west. You know, does AI get out of our human You know, does AI get out of our human You know, does AI get out of our human control? It's a long-term concern in control? It's a long-term concern in control? It's a long-term concern in China, but one that's taken a bunch of China, but one that's taken a bunch of China, but one that's taken a bunch of different forms. You know, what do they different forms. You know, what do they different forms. You know, what do they mean? Are they talking about party mean? Are they talking about party mean? Are they talking about party control? Are they talking about, you control? Are they talking about, you control? Are they talking about, you know, the control of an operator? Or are know, the control of an operator? Or are know, the control of an operator? Or are they talking about human control over they talking about human control over they talking about human control over AI? And so he put down a marker there AI? And so he put down a marker there AI? And so he put down a marker there around loss of control. And I read it as around loss of control. And I read it as around loss of control. And I read it as leaving this space open. These are all leaving this space open. These are all leaving this space open. These are all signals to people in the system. When he signals to people in the system. When he signals to people in the system. When he says forestall loss of control, that says forestall loss of control, that says forestall loss of control, that means that AI researchers all throughout means that AI researchers all throughout means that AI researchers all throughout China, when they're applying for the China, when they're applying for the China, when they're applying for the next grant, they're going to use that next grant, they're going to use that next grant, they're going to use that term. If you use a term that was in a term. If you use a term that was in a term. If you use a term that was in a big she speech, you're just more likely big she speech, you're just more likely big she speech, you're just more likely to get grant funding. Like these things to get grant funding. Like these things to get grant funding. Like these things are markers that everyone, the policy are markers that everyone, the policy are markers that everyone, the policy makers are interpreting and they're makers are interpreting and they're makers are interpreting and they're trying to figure out how can I do that trying to figure out how can I do that trying to figure out how can I do that in my area. researchers looking for in my area. researchers looking for in my area. researchers looking for funding are adopting it. Companies are funding are adopting it. Companies are funding are adopting it. Companies are looking for signals about what will and looking for signals about what will and looking for signals about what will and won't be sort of, you know, inbounds.

won't be sort of, you know, inbounds. won't be sort of, you know, inbounds. So, the words really matter. And I think So, the words really matter. And I think So, the words really matter. And I think that conclusion was at least putting that conclusion was at least putting that conclusion was at least putting down some markers that are that are down some markers that are that are down some markers that are that are showing that China is shifting pretty showing that China is shifting pretty showing that China is shifting pretty quickly on a couple of these fronts. My quickly on a couple of these fronts. My quickly on a couple of these fronts. My model of this is that political model of this is that political model of this is that political pressure, political possibility pressure, political possibility pressure, political possibility doesn't build linearly, and it pically doesn't build linearly, and it pically doesn't build linearly, and it pically will not on AI. that what happens is you will not on AI. that what happens is you will not on AI. that what happens is you have issues, they stagnate, they are not have issues, they stagnate, they are not have issues, they stagnate, they are not at the front of the agenda at the front of the agenda at the front of the agenda and then something happens and then something happens and then something happens and the window of possibility blows and the window of possibility blows and the window of possibility blows open. So I think in America here the open. So I think in America here the open. So I think in America here the open AI hugging face hacks the I mean open AI hugging face hacks the I mean open AI hugging face hacks the I mean almost more consequentially the fact almost more consequentially the fact almost more consequentially the fact that OpenAI systems hacked open AI and that OpenAI systems hacked open AI and that OpenAI systems hacked open AI and took over part of their research took over part of their research took over part of their research clusters. uh the kind of social clusters. uh the kind of social clusters. uh the kind of social engineering and effort to upload engineering and effort to upload engineering and effort to upload malicious code from you know frontier malicious code from you know frontier malicious code from you know frontier anthropic models that the sort of swarm anthropic models that the sort of swarm anthropic models that the sort of swarm behavior, the peer behavior that that behavior, the peer behavior that that behavior, the peer behavior that that this summer of weird AI incidents this summer of weird AI incidents this summer of weird AI incidents has blown this open a bit in America and has blown this open a bit in America and has blown this open a bit in America and now all of a sudden we're talking about now all of a sudden we're talking about now all of a sudden we're talking about pacing the frontier and pacing the frontier and pacing the frontier and and so I guess a question is China knows and so I guess a question is China knows and so I guess a question is China knows these things are happening so how are these things are happening so how are these things are happening so how are they respond responding to these same they respond responding to these same they respond responding to these same events that are transforming our events that are transforming our events that are transforming our conversation.

conversation. conversation. >> So, you're right that they're taking it >> So, you're right that they're taking it >> So, you're right that they're taking it in. There's tons of coverage in Chinese in. There's tons of coverage in Chinese in. There's tons of coverage in Chinese state media about the Hugging Face state media about the Hugging Face state media about the Hugging Face incident about pretty much all the major incident about pretty much all the major incident about pretty much all the major safety developments. You know, recently safety developments. You know, recently safety developments. You know, recently there was an anthropic researcher who there was an anthropic researcher who there was an anthropic researcher who resigned and who issued these pretty resigned and who issued these pretty resigned and who issued these pretty dire warnings like that was in state dire warnings like that was in state dire warnings like that was in state media today. I was reading that. And so, media today. I was reading that. And so, media today. I was reading that. And so, they they take it all in. they they are they they take it all in. they they are they they take it all in. they they are much more attuned to our conversation much more attuned to our conversation much more attuned to our conversation than we are to theirs. I think part of than we are to theirs. I think part of than we are to theirs. I think part of it is they they react more incrementally it is they they react more incrementally it is they they react more incrementally than we do. And that's in part, I think, than we do. And that's in part, I think, than we do. And that's in part, I think, due to this kind of long-term different due to this kind of long-term different due to this kind of long-term different relationship to say super intelligence relationship to say super intelligence relationship to say super intelligence and catastrophic risk. For a lot of and catastrophic risk. For a lot of and catastrophic risk. For a lot of Americans who have been thinking about Americans who have been thinking about Americans who have been thinking about this for, you know, a decade, this this for, you know, a decade, this this for, you know, a decade, this they've been predicting this will happen they've been predicting this will happen they've been predicting this will happen and then this happened and it's the it and then this happened and it's the it and then this happened and it's the it is the ultimate illustration that they is the ultimate illustration that they is the ultimate illustration that they were right all along and it's happening. were right all along and it's happening. were right all along and it's happening. >> For the Chinese side, this is just >> For the Chinese side, this is just >> For the Chinese side, this is just newer. They're taking it on board and newer. They're taking it on board and newer. They're taking it on board and they take the data point. It's like, they take the data point. It's like, they take the data point. It's like, okay, that's interesting. It hacked out okay, that's interesting. It hacked out okay, that's interesting. It hacked out of a system. Like, was this a issue with of a system. Like, was this a issue with of a system. Like, was this a issue with the safeguards, with the tooling? Could the safeguards, with the tooling? Could the safeguards, with the tooling? Could this have been, you know, constrained uh this have been, you know, constrained uh this have been, you know, constrained uh with pretty like mundane security with pretty like mundane security with pretty like mundane security measures or is this a sign of something measures or is this a sign of something measures or is this a sign of something bigger? And it's it's been really bigger? And it's it's been really bigger? And it's it's been really interesting to watch over the last year, interesting to watch over the last year, interesting to watch over the last year, really year and a half at this point as really year and a half at this point as really year and a half at this point as a lot of this safety terminology has a lot of this safety terminology has a lot of this safety terminology has worked its way into important Chinese worked its way into important Chinese worked its way into important Chinese government documents, important government documents, important government documents, important technical standards, documents by their technical standards, documents by their technical standards, documents by their lead regulator. I think just last week, lead regulator. I think just last week, lead regulator. I think just last week, China's main AI regulator, the cyerspace

China's main AI regulator, the cyerspace China's main AI regulator, the cyerspace administration of China, issued uh sort administration of China, issued uh sort administration of China, issued uh sort of a public statement on its top five AI of a public statement on its top five AI of a public statement on its top five AI risks. And number two on that list risks. And number two on that list risks. And number two on that list included what they call extreme loss of included what they call extreme loss of included what they call extreme loss of control, Zidwan Shukong. That's the control, Zidwan Shukong. That's the control, Zidwan Shukong. That's the first time that I've seen that specific first time that I've seen that specific first time that I've seen that specific phrase, extreme loss of control. They'll phrase, extreme loss of control. They'll phrase, extreme loss of control. They'll talk about controllability. In the past talk about controllability. In the past talk about controllability. In the past when they talked about that that was when they talked about that that was when they talked about that that was more like party state controllability more like party state controllability more like party state controllability but around 2021 and then really in 2023 but around 2021 and then really in 2023 but around 2021 and then really in 2023 they started talking about human control they started talking about human control they started talking about human control over AI and now it's it's essentially over AI and now it's it's essentially over AI and now it's it's essentially it's working its way into more and more it's working its way into more and more it's working its way into more and more practical and specific AI policy and practical and specific AI policy and practical and specific AI policy and technical documents. And so they're technical documents. And so they're technical documents. And so they're taking it on board. I think they are you taking it on board. I think they are you taking it on board. I think they are you know essentially they are moving in the know essentially they are moving in the know essentially they are moving in the right direction on a lot of this. And to right direction on a lot of this. And to right direction on a lot of this. And to me the big open question is like do they me the big open question is like do they me the big open question is like do they move fast enough move fast enough move fast enough [music] there was another incident that it there was another incident that it didn't I mean it literally made didn't I mean it literally made didn't I mean it literally made headlines here but it has not been headlines here but it has not been headlines here but it has not been greeted as such a big deal here even greeted as such a big deal here even greeted as such a big deal here even though to me it was quite scary. So, though to me it was quite scary. So, though to me it was quite scary. So, Frontier AI models were able to find a Frontier AI models were able to find a Frontier AI models were able to find a vulnerability in WeChat, which maybe you vulnerability in WeChat, which maybe you vulnerability in WeChat, which maybe you can describe for an American audience, can describe for an American audience, can describe for an American audience, the centrality of WeChat to the Chinese the centrality of WeChat to the Chinese the centrality of WeChat to the Chinese digital ecosystem, digital ecosystem, digital ecosystem, and they're able to build this attack on

and they're able to build this attack on and they're able to build this attack on it that they dubbed WiiWorm. And it it that they dubbed WiiWorm. And it it that they dubbed WiiWorm. And it would have given control of somebody's would have given control of somebody's would have given control of somebody's phone just by calling that phone. phone just by calling that phone. phone just by calling that phone. Now this was then conveyed to Tencent Now this was then conveyed to Tencent Now this was then conveyed to Tencent the developers of WeChat and the the developers of WeChat and the the developers of WeChat and the according to them the vulnerability has according to them the vulnerability has according to them the vulnerability has been patched but it seemed like a hell been patched but it seemed like a hell been patched but it seemed like a hell of an example to China that the hacking of an example to China that the hacking of an example to China that the hacking capabilities here at the point where it capabilities here at the point where it capabilities here at the point where it could compromise major foundational could compromise major foundational could compromise major foundational Chinese digital infrastructure. Chinese digital infrastructure. Chinese digital infrastructure. How has that been covered? I haven't How has that been covered? I haven't How has that been covered? I haven't seen that much coverage of it in seen that much coverage of it in seen that much coverage of it in mainstream media. And that might be mainstream media. And that might be mainstream media. And that might be because when an American company finds a because when an American company finds a because when an American company finds a huge vulnerability in like the central, huge vulnerability in like the central, huge vulnerability in like the central, you know, digital platform in China, you know, digital platform in China, you know, digital platform in China, it's not really seen as in everybody's it's not really seen as in everybody's it's not really seen as in everybody's interest to publicize that a ton. So interest to publicize that a ton. So interest to publicize that a ton. So that might be part of it. Um, I think that might be part of it. Um, I think that might be part of it. Um, I think another part of it is that this is a another part of it is that this is a another part of it is that this is a question of like offensive hacking question of like offensive hacking question of like offensive hacking capabilities. And I think for them like capabilities. And I think for them like capabilities. And I think for them like the real wakeup moment for that came the real wakeup moment for that came the real wakeup moment for that came with mythos when they you know the US with mythos when they you know the US with mythos when they you know the US develops a system that they're not develops a system that they're not develops a system that they're not releasing to the public that they're releasing to the public that they're releasing to the public that they're only releasing to a set number of only releasing to a set number of only releasing to a set number of companies and also the NSA and China has companies and also the NSA and China has companies and also the NSA and China has to assume at that point that it is being to assume at that point that it is being to assume at that point that it is being deployed you know far and wide against deployed you know far and wide against deployed you know far and wide against Chinese systems. So a lot of the Chinese systems. So a lot of the Chinese systems. So a lot of the discussion in the aftermath of that was discussion in the aftermath of that was discussion in the aftermath of that was about how do we harden our own system about how do we harden our own system about how do we harden our own system against these type of cyber attacks. You against these type of cyber attacks. You against these type of cyber attacks. You know, in some sense, this type of cyber know, in some sense, this type of cyber know, in some sense, this type of cyber warfare between the two countries is warfare between the two countries is warfare between the two countries is inevitable and long term. And like it's inevitable and long term. And like it's inevitable and long term. And like it's almost like we shouldn't take it too

almost like we shouldn't take it too almost like we shouldn't take it too personally. China shouldn't take it too personally. China shouldn't take it too personally. China shouldn't take it too personally when we, you know, hack them personally when we, you know, hack them personally when we, you know, hack them in a bunch of ways. We shouldn't take it in a bunch of ways. We shouldn't take it in a bunch of ways. We shouldn't take it too personally when they didn't. That's too personally when they didn't. That's too personally when they didn't. That's that's kind of our job and their job. that's kind of our job and their job. that's kind of our job and their job. >> It's >> It's >> It's it's the it's the question of like when it's the it's the question of like when it's the it's the question of like when something happens, something happens, something happens, >> a lot a lot caught up in there. >> a lot a lot caught up in there. >> a lot a lot caught up in there. [laughter] I'm not going to question it, [laughter] I'm not going to question it, [laughter] I'm not going to question it, but just just putting a pin that uh but just just putting a pin that uh but just just putting a pin that uh >> yeah, a lot it's it's the job of the NSA >> yeah, a lot it's it's the job of the NSA >> yeah, a lot it's it's the job of the NSA and it is the job of the MSS to try to and it is the job of the MSS to try to and it is the job of the MSS to try to hack each other. There should be limits, hack each other. There should be limits, hack each other. There should be limits, you know, critical infrastructure, all you know, critical infrastructure, all you know, critical infrastructure, all that kind of stuff. But in some ways, I that kind of stuff. But in some ways, I that kind of stuff. But in some ways, I think that's baked into both countries think that's baked into both countries think that's baked into both countries worldview that we're both going to be worldview that we're both going to be worldview that we're both going to be using it intentionally against each using it intentionally against each using it intentionally against each other. The issue is when it's something other. The issue is when it's something other. The issue is when it's something that's not being done intentionally by a that's not being done intentionally by a that's not being done intentionally by a state. When it's happening by a state. When it's happening by a state. When it's happening by a non-state actor that neither of us wants non-state actor that neither of us wants non-state actor that neither of us wants these tools in the hands of when it's these tools in the hands of when it's these tools in the hands of when it's out of control and neither of us has the out of control and neither of us has the out of control and neither of us has the ability to sort of understand or control ability to sort of understand or control ability to sort of understand or control it. Those are kind of the areas where I it. Those are kind of the areas where I it. Those are kind of the areas where I would expect, you know, some level of would expect, you know, some level of would expect, you know, some level of overlap and it's that's that's a newer overlap and it's that's that's a newer overlap and it's that's that's a newer phenomenon that China's grappling with. phenomenon that China's grappling with. phenomenon that China's grappling with. What about recursive self-improvement?

What about recursive self-improvement? What about recursive self-improvement? Sometimes we talk about loss of control Sometimes we talk about loss of control Sometimes we talk about loss of control like it is a passive thing. Uh, I don't like it is a passive thing. Uh, I don't like it is a passive thing. Uh, I don't intend to lose my keys, but I do it all intend to lose my keys, but I do it all intend to lose my keys, but I do it all the time. Loss of keys. [sighs] the time. Loss of keys. [sighs] the time. Loss of keys. [sighs] Recursive self-improvement is handing of Recursive self-improvement is handing of Recursive self-improvement is handing of control over to AIS, right? Recursive control over to AIS, right? Recursive control over to AIS, right? Recursive self-improvement is where the AI systems self-improvement is where the AI systems self-improvement is where the AI systems autonomously build the next system, autonomously build the next system, autonomously build the next system, right? That they're now moving faster right? That they're now moving faster right? That they're now moving faster and improvements than human beings can and improvements than human beings can and improvements than human beings can possibly keep up with. We are dependent possibly keep up with. We are dependent possibly keep up with. We are dependent on the AI system to tell us what it is on the AI system to tell us what it is on the AI system to tell us what it is doing. um we are dependent on those doing. um we are dependent on those doing. um we are dependent on those descriptions of what is happening being descriptions of what is happening being descriptions of what is happening being correct. Um we have seen AI exhibiting correct. Um we have seen AI exhibiting correct. Um we have seen AI exhibiting deceptive behavior. We see the frontier deceptive behavior. We see the frontier deceptive behavior. We see the frontier lab saying our capacity to monitor is lab saying our capacity to monitor is lab saying our capacity to monitor is already degrading. We are seeing AI labs already degrading. We are seeing AI labs already degrading. We are seeing AI labs that are currently racing towards that are currently racing towards that are currently racing towards recursive self-improvement uh expressing recursive self-improvement uh expressing recursive self-improvement uh expressing very high levels of concern about what very high levels of concern about what very high levels of concern about what it will mean to achieve this thing that it will mean to achieve this thing that it will mean to achieve this thing that they are desperately trying to achieve. they are desperately trying to achieve. they are desperately trying to achieve. It is a very strange situation. And then It is a very strange situation. And then It is a very strange situation. And then of course when you say maybe you of course when you say maybe you of course when you say maybe you shouldn't do this you get but China now shouldn't do this you get but China now shouldn't do this you get but China now she also keeps talking about how AI she also keeps talking about how AI she also keeps talking about how AI should be developed by humanity should should be developed by humanity should should be developed by humanity should be under humanity's control be under humanity's control be under humanity's control recursive self-improvement is the recursive self-improvement is the recursive self-improvement is the simplest way to give up human control of simplest way to give up human control of simplest way to give up human control of AI but to me that's a place where AI but to me that's a place where AI but to me that's a place where some international standards seem really some international standards seem really some international standards seem really needed and not like we can wait 5 years needed and not like we can wait 5 years needed and not like we can wait 5 years on that because you know the I think the on that because you know the I think the on that because you know the I think the American labs think they're going to hit American labs think they're going to hit American labs think they're going to hit RSI in the next 18 months or so. Is RSI in the next 18 months or so. Is RSI in the next 18 months or so. Is there like the possibility of

there like the possibility of there like the possibility of cooperation on this or constructive cooperation on this or constructive cooperation on this or constructive dialogue on this or is something that is dialogue on this or is something that is dialogue on this or is something that is outside a crisis point not even outside a crisis point not even outside a crisis point not even plausible within the conversation. plausible within the conversation. plausible within the conversation. >> So like a lot of these concepts or >> So like a lot of these concepts or >> So like a lot of these concepts or developments RSI is somewhat newer in developments RSI is somewhat newer in developments RSI is somewhat newer in China. Like I listen to a lot of Chinese China. Like I listen to a lot of Chinese China. Like I listen to a lot of Chinese tech podcasts and they just started tech podcasts and they just started tech podcasts and they just started talking about RSI this summer like mid talking about RSI this summer like mid talking about RSI this summer like mid late summer whereas I think this has late summer whereas I think this has late summer whereas I think this has been you know in the conversation in been you know in the conversation in been you know in the conversation in Silicon Valley for much longer than that Silicon Valley for much longer than that Silicon Valley for much longer than that now they they say wow you know this is now they they say wow you know this is now they they say wow you know this is the next thing this is where this is the next thing this is where this is the next thing this is where this is where things are going because in many where things are going because in many where things are going because in many ways as as creative and innovative as ways as as creative and innovative as ways as as creative and innovative as the Chinese ecosystem is they still do a the Chinese ecosystem is they still do a the Chinese ecosystem is they still do a lot of times essentially look to Silicon lot of times essentially look to Silicon lot of times essentially look to Silicon Valley for these type of directional Valley for these type of directional Valley for these type of directional shifts. you know, what is the next shifts. you know, what is the next shifts. you know, what is the next paradigm? paradigm? paradigm? And so with so many of the US labs And so with so many of the US labs And so with so many of the US labs beating the drum on RSI and saying like beating the drum on RSI and saying like beating the drum on RSI and saying like this is where it's going, I think this is where it's going, I think this is where it's going, I think they're the Chinese labs are kind of they're the Chinese labs are kind of they're the Chinese labs are kind of following into that space. I don't think following into that space. I don't think following into that space. I don't think they have as much experience with it. I they have as much experience with it. I they have as much experience with it. I don't think they've done as much don't think they've done as much don't think they've done as much technical work with it or maybe even technical work with it or maybe even technical work with it or maybe even thought as much about the risks of it.

thought as much about the risks of it. thought as much about the risks of it. I do think that this is one of those I do think that this is one of those I do think that this is one of those places where we might just have to draw places where we might just have to draw places where we might just have to draw a line and whether it is done a line and whether it is done a line and whether it is done bilaterally at the exact same time or bilaterally at the exact same time or bilaterally at the exact same time or whether it is something done whether it is something done whether it is something done unilaterally with the expectation or unilaterally with the expectation or unilaterally with the expectation or intense negotiation to try to uh get intense negotiation to try to uh get intense negotiation to try to uh get China to agree to the same limitation. China to agree to the same limitation. China to agree to the same limitation. That might be the point. I mean this is That might be the point. I mean this is That might be the point. I mean this is a core a core a core >> would we be more likely to get them to >> would we be more likely to get them to >> would we be more likely to get them to agree to it if we drew that line agree to it if we drew that line agree to it if we drew that line unilaterally? unilaterally? unilaterally? >> Yes. you know, if we do it unilaterally, >> Yes. you know, if we do it unilaterally, >> Yes. you know, if we do it unilaterally, it increases the chances that China does it increases the chances that China does it increases the chances that China does it. You know, it also increases some it. You know, it also increases some it. You know, it also increases some risks that you do it unilaterally and risks that you do it unilaterally and risks that you do it unilaterally and then China catches up or forges ahead. then China catches up or forges ahead. then China catches up or forges ahead. So, I it's that's a double-edged sword, So, I it's that's a double-edged sword, So, I it's that's a double-edged sword, and I won't pretend that it's just like and I won't pretend that it's just like and I won't pretend that it's just like the solve all for us to do it the solve all for us to do it the solve all for us to do it unilaterally, but, you know, a lot of unilaterally, but, you know, a lot of unilaterally, but, you know, a lot of this is a matter of sending like costly this is a matter of sending like costly this is a matter of sending like costly signals. You talked about all of the signals. You talked about all of the signals. You talked about all of the misinterpretation and sort of misinterpretation and sort of misinterpretation and sort of conspiratorial thinking between the two conspiratorial thinking between the two conspiratorial thinking between the two sides. And so when we just say a bunch sides. And so when we just say a bunch sides. And so when we just say a bunch of stuff about the dangers of RSI and we of stuff about the dangers of RSI and we of stuff about the dangers of RSI and we talk about it, but we don't actually talk about it, but we don't actually talk about it, but we don't actually have any regulations, we don't do have any regulations, we don't do have any regulations, we don't do anything about it. We are not sending anything about it. We are not sending anything about it. We are not sending any costly signals. And our lead in this any costly signals. And our lead in this any costly signals. And our lead in this technology allows us to have access to technology allows us to have access to technology allows us to have access to information, to see threats, and to see information, to see threats, and to see information, to see threats, and to see risks that they just haven't seen yet.

risks that they just haven't seen yet. risks that they just haven't seen yet. They don't they're not going to trust They don't they're not going to trust They don't they're not going to trust everything that we say. They're not everything that we say. They're not everything that we say. They're not going to trust all the information that going to trust all the information that going to trust all the information that we share. But that is that's a card that we share. But that is that's a card that we share. But that is that's a card that we can play in these areas. And whether we can play in these areas. And whether we can play in these areas. And whether it's a unilateral pause or it's a an it's a unilateral pause or it's a an it's a unilateral pause or it's a an information sharing mechanism that we information sharing mechanism that we information sharing mechanism that we set up now something where we share set up now something where we share set up now something where we share information on incidents like the information on incidents like the information on incidents like the hugging face incident you know if the US hugging face incident you know if the US hugging face incident you know if the US and China are going to sit down and talk and China are going to sit down and talk and China are going to sit down and talk about AI in the coming weeks that's a about AI in the coming weeks that's a about AI in the coming weeks that's a great opportunity to put on the table a great opportunity to put on the table a great opportunity to put on the table a lot of information that's not sensitive lot of information that's not sensitive lot of information that's not sensitive in the sense that it doesn't undermine in the sense that it doesn't undermine in the sense that it doesn't undermine the US lead but it lays out very clearly the US lead but it lays out very clearly the US lead but it lays out very clearly and in deeply technical terms like this and in deeply technical terms like this and in deeply technical terms like this is what we saw and and this is why we're is what we saw and and this is why we're is what we saw and and this is why we're worried about it. Do with that what you worried about it. Do with that what you worried about it. Do with that what you want, but this is what we saw. I think want, but this is what we saw. I think want, but this is what we saw. I think that's that's the space that we want to that's that's the space that we want to that's that's the space that we want to be working in. And then we want to be be working in. And then we want to be be working in. And then we want to be other than just sharing the information other than just sharing the information other than just sharing the information on the risk, we want to be trying to on the risk, we want to be trying to on the risk, we want to be trying to plant seeds or enhance the technical AI plant seeds or enhance the technical AI plant seeds or enhance the technical AI safety capabilities within China. They safety capabilities within China. They safety capabilities within China. They they really need to catch up. The they really need to catch up. The they really need to catch up. The practices there are just much further practices there are just much further practices there are just much further behind the leading US labs. The behind the leading US labs. The behind the leading US labs. The capabilities aren't that far behind, but capabilities aren't that far behind, but capabilities aren't that far behind, but I think the safety practices are further I think the safety practices are further I think the safety practices are further behind. And so it's in our interest, you behind. And so it's in our interest, you behind. And so it's in our interest, you know, loss of control. If something goes know, loss of control. If something goes know, loss of control. If something goes out of control in China, it doesn't stop out of control in China, it doesn't stop out of control in China, it doesn't stop at the borders there. So it's in our at the borders there. So it's in our at the borders there. So it's in our interests for them to have good safety interests for them to have good safety interests for them to have good safety practices and they have a lot of practices and they have a lot of practices and they have a lot of catching up to do.

catching up to do. catching up to do. >> So we're on the cusp of there being >> So we're on the cusp of there being >> So we're on the cusp of there being talks. Um they're being led on the US talks. Um they're being led on the US talks. Um they're being led on the US side by Treasury Secretary Scott side by Treasury Secretary Scott side by Treasury Secretary Scott Bessant. He's got Chinese counterpart. Bessant. He's got Chinese counterpart. Bessant. He's got Chinese counterpart. There's also going to be the Trump and There's also going to be the Trump and There's also going to be the Trump and Xi Jinping meetings coming up. Xi Jinping meetings coming up. Xi Jinping meetings coming up. I think that uh you've sort of been I think that uh you've sort of been I think that uh you've sort of been pouring a little bit of cold water for pouring a little bit of cold water for pouring a little bit of cold water for people on what to expect out of these, people on what to expect out of these, people on what to expect out of these, but what to you is but what to you is but what to you is a constructive outcome here, right? The a constructive outcome here, right? The a constructive outcome here, right? The sort of beginning of a space in which sort of beginning of a space in which sort of beginning of a space in which you know possibilities can emerge you know possibilities can emerge you know possibilities can emerge and what to you would be and what to you would be and what to you would be you know a negative signal about what's you know a negative signal about what's you know a negative signal about what's possible. possible. possible. So I think a negative signal would be uh So I think a negative signal would be uh So I think a negative signal would be uh a statement that sounds good and a statement that sounds good and a statement that sounds good and gestures at some thing that nobody has a gestures at some thing that nobody has a gestures at some thing that nobody has a problem with. So if the two sides get problem with. So if the two sides get problem with. So if the two sides get together and they say look we both care together and they say look we both care together and they say look we both care about AI and we both care about child about AI and we both care about child about AI and we both care about child safety and so you know we each affirm safety and so you know we each affirm safety and so you know we each affirm our our commitment to child safety and our our commitment to child safety and our our commitment to child safety and AI. It's like sure yeah that that is an AI. It's like sure yeah that that is an AI. It's like sure yeah that that is an important issue but that is not kind of important issue but that is not kind of important issue but that is not kind of the key issue between the two countries.

the key issue between the two countries. the key issue between the two countries. So I think that would be a sign that we So I think that would be a sign that we So I think that would be a sign that we hadn't really we didn't really have hadn't really we didn't really have hadn't really we didn't really have traction yet at least. I think what traction yet at least. I think what traction yet at least. I think what would be positive to me is one establish would be positive to me is one establish would be positive to me is one establish this as an ongoing recurring dialogue this as an ongoing recurring dialogue this as an ongoing recurring dialogue that will have staff that will sort of that will have staff that will sort of that will have staff that will sort of build over time. So you know maybe the build over time. So you know maybe the build over time. So you know maybe the USChina strategic AI dialogue that meets USChina strategic AI dialogue that meets USChina strategic AI dialogue that meets every four months. We've in the past every four months. We've in the past every four months. We've in the past we've established these on security we've established these on security we've established these on security issues, on economic issues, and you need issues, on economic issues, and you need issues, on economic issues, and you need to have a real structure in place where to have a real structure in place where to have a real structure in place where it's not just a one-off. The next thing it's not just a one-off. The next thing it's not just a one-off. The next thing I'd like to see maybe two things. One I'd like to see maybe two things. One I'd like to see maybe two things. One would be a working group between the would be a working group between the would be a working group between the leading technical AI safety people leading technical AI safety people leading technical AI safety people within the US government and the leading within the US government and the leading within the US government and the leading technical AI safety people within the technical AI safety people within the technical AI safety people within the Chinese system. So in the US there you Chinese system. So in the US there you Chinese system. So in the US there you know there's a lot of bureaucratic know there's a lot of bureaucratic know there's a lot of bureaucratic fighting over this but the center for AI fighting over this but the center for AI fighting over this but the center for AI standards and innovation the Casey is standards and innovation the Casey is standards and innovation the Casey is really I think the center of knowledge really I think the center of knowledge really I think the center of knowledge when it comes to testing of frontier when it comes to testing of frontier when it comes to testing of frontier models. China has a new working group models. China has a new working group models. China has a new working group called working group 9 that is called working group 9 that is called working group 9 that is essentially tasked with developing essentially tasked with developing essentially tasked with developing technical standards related to AI safety technical standards related to AI safety technical standards related to AI safety broadly defined but also starting to broadly defined but also starting to broadly defined but also starting to look at catastrophic risks. I think look at catastrophic risks. I think look at catastrophic risks. I think something that creates a space where something that creates a space where something that creates a space where those two teams can safely talk to each those two teams can safely talk to each those two teams can safely talk to each other [snorts] and exchange best other [snorts] and exchange best other [snorts] and exchange best practices. Say, "This is what we're practices. Say, "This is what we're practices. Say, "This is what we're seeing. This is what we're worried seeing. This is what we're worried seeing. This is what we're worried about. This is how we test for it and about. This is how we test for it and about. This is how we test for it and this is how we mitigate it." Maybe the this is how we mitigate it." Maybe the this is how we mitigate it." Maybe the second thing would be a a crisis second thing would be a a crisis second thing would be a a crisis communication line. A good way for the communication line. A good way for the communication line. A good way for the US and China to get in touch if US and China to get in touch if US and China to get in touch if something emerges rapidly that is a something emerges rapidly that is a something emerges rapidly that is a AIdriven crisis that could get spun even AIdriven crisis that could get spun even AIdriven crisis that could get spun even further out of control because of

further out of control because of further out of control because of USChina dynamics. So you know the USChina dynamics. So you know the USChina dynamics. So you know the hugging face incident somewhat luckily hugging face incident somewhat luckily hugging face incident somewhat luckily open AAI hacked hugging face they were open AAI hacked hugging face they were open AAI hacked hugging face they were able to more or less get in touch with able to more or less get in touch with able to more or less get in touch with each other and and sort of sort it out each other and and sort of sort it out each other and and sort of sort it out and it wasn't a big deal and it wasn't a big deal and it wasn't a big deal >> and the hack was somewhat untangled by >> and the hack was somewhat untangled by >> and the hack was somewhat untangled by hugging face using Chinese openweight hugging face using Chinese openweight hugging face using Chinese openweight models models models >> Chinese models >> Chinese models >> Chinese models >> which the Chinese state media quite >> which the Chinese state media quite >> which the Chinese state media quite enjoyed. enjoyed. enjoyed. >> Absolutely. You know, imagine just a >> Absolutely. You know, imagine just a >> Absolutely. You know, imagine just a couple different like twists on that. couple different like twists on that. couple different like twists on that. Like what if that is a DeepSseek model Like what if that is a DeepSseek model Like what if that is a DeepSseek model that's hacking hugging face or what if that's hacking hugging face or what if that's hacking hugging face or what if it's an open AI model that for whatever it's an open AI model that for whatever it's an open AI model that for whatever reason decides it really needs to reason decides it really needs to reason decides it really needs to acquire more compute resources and oh I acquire more compute resources and oh I acquire more compute resources and oh I I can find this, you know, insecure I can find this, you know, insecure I can find this, you know, insecure compute cluster that happens to be in, compute cluster that happens to be in, compute cluster that happens to be in, you know, Jang province in China. What you know, Jang province in China. What you know, Jang province in China. What happens if if it takes over a compute happens if if it takes over a compute happens if if it takes over a compute cluster there? Like how is China going cluster there? Like how is China going cluster there? Like how is China going to read that signal? What's their to read that signal? What's their to read that signal? What's their response going to be? or even if you response going to be? or even if you response going to be? or even if you take it out of a just a purely bilateral take it out of a just a purely bilateral take it out of a just a purely bilateral context. If we start to get information context. If we start to get information context. If we start to get information intelligence that a swarm of AI agents intelligence that a swarm of AI agents intelligence that a swarm of AI agents is draining bank accounts in Pakistan is draining bank accounts in Pakistan is draining bank accounts in Pakistan and we don't know what their intentions and we don't know what their intentions and we don't know what their intentions are. We cannot read their communications are. We cannot read their communications are. We cannot read their communications and we cannot shut it down right away.

and we cannot shut it down right away. and we cannot shut it down right away. We need to be in touch with the other We need to be in touch with the other We need to be in touch with the other leading AI superpower on that issue. So leading AI superpower on that issue. So leading AI superpower on that issue. So a way that these two countries can get a way that these two countries can get a way that these two countries can get in touch, share information in a crisis in touch, share information in a crisis in touch, share information in a crisis situation. It's a very fraught issue. We situation. It's a very fraught issue. We situation. It's a very fraught issue. We have had a lot of these crisis have had a lot of these crisis have had a lot of these crisis communication lines on military issues communication lines on military issues communication lines on military issues and the US complaint is always the and the US complaint is always the and the US complaint is always the Chinese side doesn't pick up the phone Chinese side doesn't pick up the phone Chinese side doesn't pick up the phone when we call them. And that's a real when we call them. And that's a real when we call them. And that's a real issue. I think you know one mitigation issue. I think you know one mitigation issue. I think you know one mitigation to that is to use somewhat ironic not to that is to use somewhat ironic not to that is to use somewhat ironic not use a phone but use a fax machine use a phone but use a fax machine use a phone but use a fax machine >> like literal faxes. >> like literal faxes. >> like literal faxes. >> Literal faxes. Um, and it's it has a >> Literal faxes. Um, and it's it has a >> Literal faxes. Um, and it's it has a logic to it too because the political logic to it too because the political logic to it too because the political system there is not a system of system there is not a system of system there is not a system of empowered individuals. It's a system of empowered individuals. It's a system of empowered individuals. It's a system of committees and a system of documents. committees and a system of documents. committees and a system of documents. And so when our, you know, treasury And so when our, you know, treasury And so when our, you know, treasury secretary, someone who feels very secretary, someone who feels very secretary, someone who feels very empowered on the US side picks up the empowered on the US side picks up the empowered on the US side picks up the phone and like give me some answers, you phone and like give me some answers, you phone and like give me some answers, you know, Khalif or other Chinese know, Khalif or other Chinese know, Khalif or other Chinese counterpart counterpart counterpart not really ready to give you answers on not really ready to give you answers on not really ready to give you answers on the fly. much better to send a document the fly. much better to send a document the fly. much better to send a document over to their system that they can over to their system that they can over to their system that they can review, they can bring it to their review, they can bring it to their review, they can bring it to their committee, they can come up with their committee, they can come up with their committee, they can come up with their understanding and response and send understanding and response and send understanding and response and send something back. So something in that something back. So something in that something back. So something in that vein that at least puts a little bit of vein that at least puts a little bit of vein that at least puts a little bit of a safety net on these incidents that I a safety net on these incidents that I a safety net on these incidents that I think you know it's pretty like think you know it's pretty like think you know it's pretty like something like that is pretty likely to something like that is pretty likely to something like that is pretty likely to happen in the next year. Everything happen in the next year. Everything happen in the next year. Everything you're saying here makes sense to me you're saying here makes sense to me you're saying here makes sense to me about the facts not phone dynamic and about the facts not phone dynamic and about the facts not phone dynamic and documents and [gasps] documents and [gasps] documents and [gasps] but man when the whole thing we're but man when the whole thing we're but man when the whole thing we're facing down is the acceleration of AI facing down is the acceleration of AI facing down is the acceleration of AI incidents and things happening at faster

incidents and things happening at faster incidents and things happening at faster than human speeds and now we're dealing than human speeds and now we're dealing than human speeds and now we're dealing with governments that work at frankly with governments that work at frankly with governments that work at frankly slower than human speeds. slower than human speeds. slower than human speeds. It really creates quite a mismatch. I It really creates quite a mismatch. I It really creates quite a mismatch. I mean you know the the big language right mean you know the the big language right mean you know the the big language right now is pacing the frontier but we don't now is pacing the frontier but we don't now is pacing the frontier but we don't even have a plan to keep the frontier even have a plan to keep the frontier even have a plan to keep the frontier from accelerating. from accelerating. from accelerating. forget pacing it at the moment. We are forget pacing it at the moment. We are forget pacing it at the moment. We are currently accelerating the frontier. currently accelerating the frontier. currently accelerating the frontier. It's concerning. It's concerning. It's concerning. >> It's deep. It's deeply concerning. And >> It's deep. It's deeply concerning. And >> It's deep. It's deeply concerning. And if it's a matter of a race in decision-m if it's a matter of a race in decision-m if it's a matter of a race in decision-m between an agent and a person, like the between an agent and a person, like the between an agent and a person, like the agent is going to win that race. I agent is going to win that race. I agent is going to win that race. I think, you know, we hopefully won't be think, you know, we hopefully won't be think, you know, we hopefully won't be engaged in that very specific race. And engaged in that very specific race. And engaged in that very specific race. And I think the Chinese system, it's it's I think the Chinese system, it's it's I think the Chinese system, it's it's interesting because it's in some ways it interesting because it's in some ways it interesting because it's in some ways it can be so slow and incremental and it can be so slow and incremental and it can be so slow and incremental and it can be so fast. You know, their response can be so fast. You know, their response can be so fast. You know, their response to COVID was initially so halting. It to COVID was initially so halting. It to COVID was initially so halting. It was it was screwed up by information was it was screwed up by information was it was screwed up by information gaps where the local officials don't gaps where the local officials don't gaps where the local officials don't want to report the bad news to the want to report the bad news to the want to report the bad news to the higher officials. It has all these kind higher officials. It has all these kind higher officials. It has all these kind of, you know, neurosis and of, you know, neurosis and of, you know, neurosis and idiosyncrasies. But when they decide idiosyncrasies. But when they decide idiosyncrasies. But when they decide like we need to shut down this city, we like we need to shut down this city, we like we need to shut down this city, we need a wall in this city and not let need a wall in this city and not let need a wall in this city and not let anybody in or out. That happens pretty anybody in or out. That happens pretty anybody in or out. That happens pretty fast. And I'm not saying that's the fast. And I'm not saying that's the fast. And I'm not saying that's the solution on AI. That had a ton of human solution on AI. That had a ton of human solution on AI. That had a ton of human costs. It always when China takes costs. It always when China takes costs. It always when China takes actions like that, it always has a ton actions like that, it always has a ton actions like that, it always has a ton of human costs. But each side kind of of human costs. But each side kind of of human costs. But each side kind of has its strengths and weaknesses in this has its strengths and weaknesses in this has its strengths and weaknesses in this area and I think there is a chance that area and I think there is a chance that area and I think there is a chance that while China is moving pretty while China is moving pretty while China is moving pretty incrementally now as the evidence incrementally now as the evidence incrementally now as the evidence builds, I think there is a chance that builds, I think there is a chance that builds, I think there is a chance that they they shift gears pretty quickly.

they they shift gears pretty quickly. they they shift gears pretty quickly. What is the relational context between What is the relational context between What is the relational context between the leadership that these conversations the leadership that these conversations the leadership that these conversations are coming into? And you know, Trump are coming into? And you know, Trump are coming into? And you know, Trump rose in politics with a very skeptical, rose in politics with a very skeptical, rose in politics with a very skeptical, to say the least, take on China. Um, in to say the least, take on China. Um, in to say the least, take on China. Um, in his second term, after the beginning his second term, after the beginning his second term, after the beginning tariffs and liberation day, they tried tariffs and liberation day, they tried tariffs and liberation day, they tried to pivot toward trade war with China. to pivot toward trade war with China. to pivot toward trade war with China. China fought back. We functionally China fought back. We functionally China fought back. We functionally backed down. backed down. backed down. And since then, for all the bluster you And since then, for all the bluster you And since then, for all the bluster you sometimes hear, Trump seems to be trying sometimes hear, Trump seems to be trying sometimes hear, Trump seems to be trying to build a better direct relationship to build a better direct relationship to build a better direct relationship with she. And so to what degree is the with she. And so to what degree is the with she. And so to what degree is the current status of the USChina current status of the USChina current status of the USChina relationship and particularly the Trumpi relationship and particularly the Trumpi relationship and particularly the Trumpi relationship maybe more flexible relationship maybe more flexible relationship maybe more flexible than one might assume just knowing sort than one might assume just knowing sort than one might assume just knowing sort of where it was at the beginning of of where it was at the beginning of of where it was at the beginning of Trump's second term. Trump's second term. Trump's second term. >> Yeah. You know Trump does both extremes >> Yeah. You know Trump does both extremes >> Yeah. You know Trump does both extremes when it comes to China. you know, he when it comes to China. you know, he when it comes to China. you know, he totally he really changed the direction totally he really changed the direction totally he really changed the direction of American policy in a much more of American policy in a much more of American policy in a much more hawkish direction, taking like hawkish direction, taking like hawkish direction, taking like seriously, you know, harmful aggressive seriously, you know, harmful aggressive seriously, you know, harmful aggressive actions against China. [gasps] And at actions against China. [gasps] And at actions against China. [gasps] And at the same time, he seems to personally the same time, he seems to personally the same time, he seems to personally really like Xiinping. He seems to admire really like Xiinping. He seems to admire really like Xiinping. He seems to admire him. He seems to see a kindred spirit in him. He seems to see a kindred spirit in him. He seems to see a kindred spirit in some way in these two strong leaders of some way in these two strong leaders of some way in these two strong leaders of countries.

countries. countries. And I think that affects a lot of US And I think that affects a lot of US And I think that affects a lot of US policymaking. There's a lot of evidence policymaking. There's a lot of evidence policymaking. There's a lot of evidence that, you know, different potentially that, you know, different potentially that, you know, different potentially aggressive actions against China have aggressive actions against China have aggressive actions against China have been watered down because Trump doesn't been watered down because Trump doesn't been watered down because Trump doesn't want to screw things up ahead of the want to screw things up ahead of the want to screw things up ahead of the meeting. You know, during the Biden meeting. You know, during the Biden meeting. You know, during the Biden administration, I think a fair number of administration, I think a fair number of administration, I think a fair number of people were thinking sort of thinking people were thinking sort of thinking people were thinking sort of thinking ahead to saying like maybe we do need to ahead to saying like maybe we do need to ahead to saying like maybe we do need to be engaging China on AI safety. Maybe we be engaging China on AI safety. Maybe we be engaging China on AI safety. Maybe we do need to be sharing information. But do need to be sharing information. But do need to be sharing information. But they felt very constrained by the idea they felt very constrained by the idea they felt very constrained by the idea that well if if Democrats do that, if that well if if Democrats do that, if that well if if Democrats do that, if the Biden administration does that, the Biden administration does that, the Biden administration does that, we're going to get roasted as soft on we're going to get roasted as soft on we're going to get roasted as soft on China. and you know it's we're going to China. and you know it's we're going to China. and you know it's we're going to be seen as kind of you know giving away be seen as kind of you know giving away be seen as kind of you know giving away the store on AI and Trump just creates the store on AI and Trump just creates the store on AI and Trump just creates his own political gravity his own his own political gravity his own his own political gravity his own political environment where that same political environment where that same political environment where that same action action action will be read in a very different way will be read in a very different way will be read in a very different way that he doesn't have to share the same that he doesn't have to share the same that he doesn't have to share the same concerns as past administrations. So I concerns as past administrations. So I concerns as past administrations. So I think that's that's a dynamic on the think that's that's a dynamic on the think that's that's a dynamic on the Chinese side. I think she is a uh much Chinese side. I think she is a uh much Chinese side. I think she is a uh much more systematic more systematic more systematic um [clears throat] thinker and a much um [clears throat] thinker and a much um [clears throat] thinker and a much less uh much less reliant on these less uh much less reliant on these less uh much less reliant on these individual relationships and seeing this individual relationships and seeing this individual relationships and seeing this as a structural as a structural as a structural long-term contest between two systems, long-term contest between two systems, long-term contest between two systems, between two countries and the you know between two countries and the you know between two countries and the you know the day-to-day wavering of we love the day-to-day wavering of we love the day-to-day wavering of we love China, we hate China, we're we're China, we hate China, we're we're China, we hate China, we're we're blockading, we want to have double the blockading, we want to have double the blockading, we want to have double the investment. I don't think he sees that investment. I don't think he sees that investment. I don't think he sees that as a meaningful as a meaningful as a meaningful change in the overall trajectory between change in the overall trajectory between change in the overall trajectory between the two countries. And so, while I think the two countries. And so, while I think the two countries. And so, while I think the kind of the onetoone relationship,

the kind of the onetoone relationship, the kind of the onetoone relationship, you know, does she like Trump, not not you know, does she like Trump, not not you know, does she like Trump, not not all that relevant, but the the changes all that relevant, but the the changes all that relevant, but the the changes in the Overton window of what we think in the Overton window of what we think in the Overton window of what we think is possible when it comes to engagement, is possible when it comes to engagement, is possible when it comes to engagement, I think that is meaningful. I think that is meaningful. I think that is meaningful. >> You're a pretty calm seeming person >> You're a pretty calm seeming person >> You're a pretty calm seeming person temperamentally. If you're talking temperamentally. If you're talking temperamentally. If you're talking honestly to maybe you know Chinese honestly to maybe you know Chinese honestly to maybe you know Chinese counterparts who are counterparts who are counterparts who are regulating but not on the most profound regulating but not on the most profound regulating but not on the most profound set of risks or American counterparts set of risks or American counterparts set of risks or American counterparts who are worrying but not actually doing who are worrying but not actually doing who are worrying but not actually doing all that much. all that much. all that much. What's your real level of alarm? Like What's your real level of alarm? Like What's your real level of alarm? Like what would you tell them about the what would you tell them about the what would you tell them about the moment we're actually in? Not what you moment we're actually in? Not what you moment we're actually in? Not what you think is possible, not what you think is think is possible, not what you think is think is possible, not what you think is likely to happen in the bilateral talks, likely to happen in the bilateral talks, likely to happen in the bilateral talks, but if everybody was where you were, the but if everybody was where you were, the but if everybody was where you were, the way they would see this issue right now. way they would see this issue right now. way they would see this issue right now. I mean, I think that the moment, this I mean, I think that the moment, this I mean, I think that the moment, this period of time is is terrifying. like we period of time is is terrifying. like we period of time is is terrifying. like we should be terrified on a certain level, should be terrified on a certain level, should be terrified on a certain level, but like I, you know, I've been working but like I, you know, I've been working but like I, you know, I've been working in AI policy one way or another since in AI policy one way or another since in AI policy one way or another since about 2017 and I've been hearing these about 2017 and I've been hearing these about 2017 and I've been hearing these warnings since then and I've always warnings since then and I've always warnings since then and I've always tried to maintain some type of like uh tried to maintain some type of like uh tried to maintain some type of like uh neutrality on how real are these risks.

neutrality on how real are these risks. neutrality on how real are these risks. I'm like, you know, these scientists say I'm like, you know, these scientists say I'm like, you know, these scientists say this, these scientists say that. I'm not this, these scientists say that. I'm not this, these scientists say that. I'm not [snorts] the one to adjudicate this. I'm [snorts] the one to adjudicate this. I'm [snorts] the one to adjudicate this. I'm not going to be the one who solves it. not going to be the one who solves it. not going to be the one who solves it. So, I'm just going to try to sort of So, I'm just going to try to sort of So, I'm just going to try to sort of keep both these things in mind and and keep both these things in mind and and keep both these things in mind and and work forward from there. But it, you work forward from there. But it, you work forward from there. But it, you know, the the evidence is mounting, the know, the the evidence is mounting, the know, the the evidence is mounting, the evidence is growing that the people have evidence is growing that the people have evidence is growing that the people have been making some of the most dire been making some of the most dire been making some of the most dire warnings for the longest time that they warnings for the longest time that they warnings for the longest time that they have probably been right at least about have probably been right at least about have probably been right at least about a lot of things. And they the warnings a lot of things. And they the warnings a lot of things. And they the warnings that they are issuing are increasingly that they are issuing are increasingly that they are issuing are increasingly dire and increasingly on short dire and increasingly on short dire and increasingly on short timelines. For someone who's been timelines. For someone who's been timelines. For someone who's been looking at this for a while and has looking at this for a while and has looking at this for a while and has tried to maintain a position of like not tried to maintain a position of like not tried to maintain a position of like not panic and neutrality, it's it's very panic and neutrality, it's it's very panic and neutrality, it's it's very it's very worrying. I think that's a it's very worrying. I think that's a it's very worrying. I think that's a place to end. Always our final question. place to end. Always our final question. place to end. Always our final question. What are three books you recommend to What are three books you recommend to What are three books you recommend to the audience? the audience? the audience? >> I'll do two China books and one fun one. >> I'll do two China books and one fun one. >> I'll do two China books and one fun one. So the first China book is Country So the first China book is Country So the first China book is Country Driving by Peter Hler, New Yorker Driving by Peter Hler, New Yorker Driving by Peter Hler, New Yorker correspondent in a lot of way for people correspondent in a lot of way for people correspondent in a lot of way for people of my generation who went over there and of my generation who went over there and of my generation who went over there and lived there. He's kind of like the lived there. He's kind of like the lived there. He's kind of like the godfather. He's the guy who inspired me godfather. He's the guy who inspired me godfather. He's the guy who inspired me to become a journalist to to just like to become a journalist to to just like to become a journalist to to just like get out into the country, meet people, get out into the country, meet people, get out into the country, meet people, you know, get such incredible, beautiful you know, get such incredible, beautiful you know, get such incredible, beautiful portraits of of Chinese society at the portraits of of Chinese society at the portraits of of Chinese society at the micro level that I think we're just micro level that I think we're just micro level that I think we're just we're missing we're missing that in so we're missing we're missing that in so we're missing we're missing that in so much of policy today. And I I hope young much of policy today. And I I hope young much of policy today. And I I hope young people today will start going back over people today will start going back over people today will start going back over there and getting in the mix. We need there and getting in the mix. We need there and getting in the mix. We need that like textured understanding. So that like textured understanding. So that like textured understanding. So that's one. Another one a little bit that's one. Another one a little bit that's one. Another one a little bit more obscure. It's called From the Soil: more obscure. It's called From the Soil: more obscure. It's called From the Soil: The Foundations of Chinese Society. It's The Foundations of Chinese Society. It's The Foundations of Chinese Society. It's a book by a Chinese sociologist in the

a book by a Chinese sociologist in the a book by a Chinese sociologist in the 30s and 40s guy named Faoong who 30s and 40s guy named Faoong who 30s and 40s guy named Faoong who was trained in the west, went back to was trained in the west, went back to was trained in the west, went back to China, applied kind of western China, applied kind of western China, applied kind of western sociological paradigms to studying sociological paradigms to studying sociological paradigms to studying Chinese villages and agriculture. And Chinese villages and agriculture. And Chinese villages and agriculture. And it's just one of the most insightful it's just one of the most insightful it's just one of the most insightful books about Chinese culture. So, I'd books about Chinese culture. So, I'd books about Chinese culture. So, I'd encourage people to to seek that one encourage people to to seek that one encourage people to to seek that one out. I read it every 3 or four years. out. I read it every 3 or four years. out. I read it every 3 or four years. And the last one, just for fun, uh And the last one, just for fun, uh And the last one, just for fun, uh Zadeie Smith's On Beauty. You know, you Zadeie Smith's On Beauty. You know, you Zadeie Smith's On Beauty. You know, you had Zadeie on the show. I just I think had Zadeie on the show. I just I think had Zadeie on the show. I just I think she's the goat. I think she's the best. she's the goat. I think she's the best. she's the goat. I think she's the best. And On Beauty is uh just a hilarious And On Beauty is uh just a hilarious And On Beauty is uh just a hilarious novel of an academic family. And um you novel of an academic family. And um you novel of an academic family. And um you know, her ability to to pierce into the know, her ability to to pierce into the know, her ability to to pierce into the psychology and the insecurities of of psychology and the insecurities of of psychology and the insecurities of of each of us [music] and and put that on each of us [music] and and put that on each of us [music] and and put that on blast in a way is just I don't know, it blast in a way is just I don't know, it blast in a way is just I don't know, it just brings me a lot of joy. So on just brings me a lot of joy. So on just brings me a lot of joy. So on beauty, Matt Shan, thank you very much. beauty, Matt Shan, thank you very much. beauty, Matt Shan, thank you very much. Thanks for having me. Thanks for having me. Thanks for having me. [music]