AMD Says 2 Ryzen AI Halos Can Run a 400B Model... I Tested It

You might have seen this device before. You might have seen this device before. This is the AMD Ryzen AI Halo. Shares This is the AMD Ryzen AI Halo. Shares This is the AMD Ryzen AI Halo. Shares the name with a Strix Halo chip that's the name with a Strix Halo chip that's the name with a Strix Halo chip that's inside there. It's an APU, so it's got inside there. It's an APU, so it's got inside there. It's an APU, so it's got the CPU, the GPU, with 128 GB of memory the CPU, the GPU, with 128 GB of memory the CPU, the GPU, with 128 GB of memory that can be shared between the two to do that can be shared between the two to do that can be shared between the two to do AI work. In my previous video, I already AI work. In my previous video, I already AI work. In my previous video, I already covered the details about this and I ran covered the details about this and I ran covered the details about this and I ran a bunch of benchmarks comparing it a bunch of benchmarks comparing it a bunch of benchmarks comparing it [music] to the Nvidia DGX Spark as well [music] to the Nvidia DGX Spark as well [music] to the Nvidia DGX Spark as well as the previous machines that had this as the previous machines that had this as the previous machines that had this chip in it. But, this one is the chip in it. But, this one is the chip in it. But, this one is the smallest. And according to AMD on a tiny smallest. And according to AMD on a tiny smallest. And according to AMD on a tiny box like this that you can fit into your box like this that you can fit into your box like this that you can fit into your pocket, well, maybe, you can run models pocket, well, maybe, you can run models pocket, well, maybe, you can run models that up to 200 billion parameters in that up to 200 billion parameters in that up to 200 billion parameters in size. In the last year that this chip size. In the last year that this chip size. In the last year that this chip has been out, AMD has done a lot of work has been out, AMD has done a lot of work has been out, AMD has done a lot of work to not only get support going for it, to not only get support going for it, to not only get support going for it, but they also came out with these but they also came out with these but they also came out with these playbooks that you can easily follow playbooks that you can easily follow playbooks that you can easily follow along to generate images, videos, text, along to generate images, videos, text, along to generate images, videos, text, use them as coding assistant, something use them as coding assistant, something use them as coding assistant, something you'd all be interested in probably. And you'd all be interested in probably. And you'd all be interested in probably. And in this advanced section, you can now in this advanced section, you can now in this advanced section, you can now cluster two of them. So, you'll be able cluster two of them. So, you'll be able cluster two of them. So, you'll be able to potentially run models that are about to potentially run models that are about to potentially run models that are about 400 billion parameters. two computers 400 billion parameters. two computers 400 billion parameters. two computers don't automatically become one larger don't automatically become one larger don't automatically become one larger computer. The model must be split computer. The model must be split computer. The model must be split somehow. The GPUs that are processing somehow. The GPUs that are processing somehow. The GPUs that are processing that data must continually exchange data that data must continually exchange data that data must continually exchange data while generating. And on this machine, while generating. And on this machine, while generating. And on this machine, that communication happens over Ethernet that communication happens over Ethernet that communication happens over Ethernet cuz this 10-gigabit port is the only cuz this 10-gigabit port is the only cuz this 10-gigabit port is the only thing we have to connect them to. So, thing we have to connect them to. So, thing we have to connect them to. So, the question is, can these two actually the question is, can these two actually the question is, can these two actually function as one practical AI system or function as one practical AI system or function as one practical AI system or will the network between them ruin the will the network between them ruin the will the network between them ruin the entire idea? According to AMD's entire idea? According to AMD's entire idea? According to AMD's playbooks here, you can use two playbooks here, you can use two playbooks here, you can use two different methods to cluster them different methods to cluster them different methods to cluster them together. One is using Llama.cpp with together. One is using Llama.cpp with together. One is using Llama.cpp with RPC, and that's basically remote

RPC, and that's basically remote RPC, and that's basically remote procedure call. And Llama.cpp is a very procedure call. And Llama.cpp is a very procedure call. And Llama.cpp is a very popular tool. O Llama, if you've heard popular tool. O Llama, if you've heard popular tool. O Llama, if you've heard of that, LM Studio, all use Llama.cpp of that, LM Studio, all use Llama.cpp of that, LM Studio, all use Llama.cpp under the hood. And you can run under the hood. And you can run under the hood. And you can run Llama.cpp directly, which is what this Llama.cpp directly, which is what this Llama.cpp directly, which is what this will do. Well, it's got RPC built into will do. Well, it's got RPC built into will do. Well, it's got RPC built into it. We'll go into that. And Rickel, it. We'll go into that. And Rickel, it. We'll go into that. And Rickel, RCCL, it's pronounced Rickel. I don't RCCL, it's pronounced Rickel. I don't RCCL, it's pronounced Rickel. I don't know. It's kind of like Nvidia has know. It's kind of like Nvidia has know. It's kind of like Nvidia has Nickel, NCCL, so this is Rickel. Rickel Nickel, NCCL, so this is Rickel. Rickel Nickel, NCCL, so this is Rickel. Rickel uses vLLM and Ray, another tool KPU's a uses vLLM and Ray, another tool KPU's a uses vLLM and Ray, another tool KPU's a of of of with tensor parallelism. Basically, it's with tensor parallelism. Basically, it's with tensor parallelism. Basically, it's a way to split up the model. I've a way to split up the model. I've a way to split up the model. I've discussed this in many videos in the discussed this in many videos in the discussed this in many videos in the past. I won't go into the details of past. I won't go into the details of past. I won't go into the details of that, but this will use that technology. that, but this will use that technology. that, but this will use that technology. So, there's two different ways of doing So, there's two different ways of doing So, there's two different ways of doing it, and both methods combine the 128 GB it, and both methods combine the 128 GB it, and both methods combine the 128 GB here and 128 GB here to give you 256 GB here and 128 GB here to give you 256 GB here and 128 GB here to give you 256 GB of available memory on the two machines. of available memory on the two machines. of available memory on the two machines. It's actually a little bit less than It's actually a little bit less than It's actually a little bit less than that because you still need a little bit that because you still need a little bit that because you still need a little bit of memory for the operating system, but of memory for the operating system, but of memory for the operating system, but both of these methods call [music] for both of these methods call [music] for both of these methods call [music] for proper 10 gigabit Ethernet network. proper 10 gigabit Ethernet network. proper 10 gigabit Ethernet network. However, you can't just take a cable However, you can't just take a cable However, you can't just take a cable >> [music] >> [music] >> [music] >> and plug them in together like that and >> and plug them in together like that and >> and plug them in together like that and have that work. That's just not going to have that work. That's just not going to have that work. That's just not going to do it. Even though each one of these has do it. Even though each one of these has do it. Even though each one of these has a 10 gig port. AMD's instructions for a 10 gig port. AMD's instructions for a 10 gig port. AMD's instructions for both of these require a 10 gigabit both of these require a 10 gigabit both of these require a 10 gigabit Ethernet switch to be used between the Ethernet switch to be used between the Ethernet switch to be used between the machines. So, yeah.

machines. So, yeah. machines. So, yeah. That's not going to work. So, before I That's not going to work. So, before I That's not going to work. So, before I can build this cluster, I need to sort can build this cluster, I need to sort can build this cluster, I need to sort out that network switch between them. out that network switch between them. out that network switch between them. These Ryzen Halo machines are now These Ryzen Halo machines are now These Ryzen Halo machines are now available at the Micro Center, which is available at the Micro Center, which is available at the Micro Center, which is their exclusive retailer. their exclusive retailer. their exclusive retailer. >> [music] >> This is Dan, by the way. You've met Dan >> This is Dan, by the way. You've met Dan before in previous videos. before in previous videos. before in previous videos. >> Nice to meet you guys. >> Nice to meet you guys. >> Nice to meet you guys. >> Say something about his hair. >> Say something about his hair. >> Say something about his hair. >> Yeah, please do. I changed it up this >> Yeah, please do. I changed it up this >> Yeah, please do. I changed it up this time. time. time. >> Dan, take us to the switches if you >> Dan, take us to the switches if you >> Dan, take us to the switches if you don't mind. don't mind. don't mind. >> I don't mind at all. Probably going to >> I don't mind at all. Probably going to >> I don't mind at all. Probably going to be in a little bit of a fancier spot. In be in a little bit of a fancier spot. In be in a little bit of a fancier spot. In theory, this is the switch area. In theory, this is the switch area. In theory, this is the switch area. In practice, like I said, I think the practice, like I said, I think the practice, like I said, I think the switch you want is probably going to be switch you want is probably going to be switch you want is probably going to be hidden somewhere. I can make it appear. hidden somewhere. I can make it appear. hidden somewhere. I can make it appear. >> Let's make it appear. Movie magic. >> Let's make it appear. Movie magic. >> Let's make it appear. Movie magic. >> I know where it is. >> I know where it is. >> I know where it is. >> Oh, you do? >> Oh, you do? >> Oh, you do? >> Yeah. >> Yeah. >> Yeah. >> What? When are you submitting your >> What? When are you submitting your >> What? When are you submitting your application? application? application? >> Dude, it's in the UniFi section. >> Dude, it's in the UniFi section. >> Dude, it's in the UniFi section. >> Yeah, but I mean, I figured, but I >> Yeah, but I mean, I figured, but I >> Yeah, but I mean, I figured, but I thought it was going to be like one of thought it was going to be like one of thought it was going to be like one of those boxes that we hide up in another those boxes that we hide up in another those boxes that we hide up in another aisle. aisle. aisle. >> But, we'll find it. Don't worry. >> But, we'll find it. Don't worry. >> But, we'll find it. Don't worry. >> We will find it. Movie magic is >> We will find it. Movie magic is >> We will find it. Movie magic is happening, I promise. happening, I promise. happening, I promise. >> In the back of the scenes. >> In the back of the scenes. >> In the back of the scenes. >> Oh, was it just out there? Oh, okay. >> Oh, was it just out there? Oh, okay. >> Oh, was it just out there? Oh, okay. >> Thank you. They put it in a different >> Thank you. They put it in a different >> Thank you. They put it in a different box now. box now. box now. >> But, it works. It's what matters, right? >> But, it works. It's what matters, right? >> But, it works. It's what matters, right? >> Do you want to like >> Do you want to like >> Do you want to like hold it?

hold it? hold it? >> See as a size comparison so they can see >> See as a size comparison so they can see >> See as a size comparison so they can see that I'm actually bigger than the box. that I'm actually bigger than the box. that I'm actually bigger than the box. >> You are bigger than the box. You >> You are bigger than the box. You >> You are bigger than the box. You couldn't fit in that box. couldn't fit in that box. couldn't fit in that box. >> I could not. >> I could not. >> I could not. >> This is that Flex XG switch that I >> This is that Flex XG switch that I >> This is that Flex XG switch that I picked up from Micro Center. picked up from Micro Center. picked up from Micro Center. But actually, it's just an empty box. I But actually, it's just an empty box. I But actually, it's just an empty box. I picked that up last year and picked that up last year and picked that up last year and I already had it. It's right here. Thing I already had it. It's right here. Thing I already had it. It's right here. Thing is, back then I didn't understand why it is, back then I didn't understand why it is, back then I didn't understand why it had four 10 gigabit ports and one 1 had four 10 gigabit ports and one 1 had four 10 gigabit ports and one 1 gigabit port. Well, now I do. It's gigabit port. Well, now I do. It's gigabit port. Well, now I do. It's actually perfect for clustering because actually perfect for clustering because actually perfect for clustering because you would plug in your cluster, up to you would plug in your cluster, up to you would plug in your cluster, up to four machines, in this case we're doing four machines, in this case we're doing four machines, in this case we're doing just two into the 10 gigabit ports, and just two into the 10 gigabit ports, and just two into the 10 gigabit ports, and the 1 gigabit port, that's just going the 1 gigabit port, that's just going the 1 gigabit port, that's just going out to your network, so you can manage out to your network, so you can manage out to your network, so you can manage your cluster. Plus, it's completely your cluster. Plus, it's completely your cluster. Plus, it's completely silent because there's no active cooling silent because there's no active cooling silent because there's no active cooling here. It's all a giant heat sink. It's here. It's all a giant heat sink. It's here. It's all a giant heat sink. It's pretty heavy, too. That would take care pretty heavy, too. That would take care pretty heavy, too. That would take care of the network, but there's still one of the network, but there's still one of the network, but there's still one little issue. I can't cluster these little issue. I can't cluster these little issue. I can't cluster these machines the way they are now. They machines the way they are now. They machines the way they are now. They don't match. The Halo I tested in the don't match. The Halo I tested in the don't match. The Halo I tested in the previous video came with Windows, and previous video came with Windows, and previous video came with Windows, and the second one had Linux on it. AMD's the second one had Linux on it. AMD's the second one had Linux on it. AMD's documentation for beginner and documentation for beginner and documentation for beginner and intermediate, all these playbooks you intermediate, all these playbooks you intermediate, all these playbooks you can do on Windows or Linux, but the can do on Windows or Linux, but the can do on Windows or Linux, but the clustering playbook is only on Linux.

clustering playbook is only on Linux. clustering playbook is only on Linux. So, I need both machines set up the same So, I need both machines set up the same So, I need both machines set up the same way. Instead of configuring the second way. Instead of configuring the second way. Instead of configuring the second machine from scratch and hoping I machine from scratch and hoping I machine from scratch and hoping I reproduced the environment exactly, I reproduced the environment exactly, I reproduced the environment exactly, I just simply cloned the Linux just simply cloned the Linux just simply cloned the Linux installation onto the Windows machine's installation onto the Windows machine's installation onto the Windows machine's SSD. That gave me two identical starting SSD. That gave me two identical starting SSD. That gave me two identical starting points with the same drivers, the same points with the same drivers, the same points with the same drivers, the same AMD software, and the same exact AMD software, and the same exact AMD software, and the same exact configuration. Of course, I knew we configuration. Of course, I knew we configuration. Of course, I knew we needed to download models onto it, so needed to download models onto it, so needed to download models onto it, so before I cloned it, I made sure that one before I cloned it, I made sure that one before I cloned it, I made sure that one of the machines had models on it, so I of the machines had models on it, so I of the machines had models on it, so I don't need to download the models twice. don't need to download the models twice. don't need to download the models twice. Also, cloning the drives also cloned the Also, cloning the drives also cloned the Also, cloning the drives also cloned the machine's identity. So, before I machine's identity. So, before I machine's identity. So, before I connected them up, I had to make sure connected them up, I had to make sure connected them up, I had to make sure that the host names were different. One that the host names were different. One that the host names were different. One became Halo 1, and one became Halo 2. became Halo 1, and one became Halo 2. became Halo 1, and one became Halo 2. >> Turn that on. >> Turn that on. >> Turn that on. >> Yes. >> Yes. >> Yes. It's Ubuntu. That used to be Windows. And we're in, baby. And we're in, baby. Nice. Now I have two matching Linux Nice. Now I have two matching Linux Nice. Now I have two matching Linux machines and yeah, I'm not going to put machines and yeah, I'm not going to put machines and yeah, I'm not going to put them on top of each other because the them on top of each other because the them on top of each other because the vents are actually on top and on the vents are actually on top and on the vents are actually on top and on the sides. So these kind of have to be sides. So these kind of have to be sides. So these kind of have to be spread apart a little bit. Just a quick spread apart a little bit. Just a quick spread apart a little bit. Just a quick little test. I got iperf3 on server two, little test. I got iperf3 on server two, little test. I got iperf3 on server two, Halo two, and iperf3 on Halo one. Just Halo two, and iperf3 on Halo one. Just Halo two, and iperf3 on Halo one. Just to test the network bandwidth between to test the network bandwidth between to test the network bandwidth between the two and it looks like it's coming the two and it looks like it's coming the two and it looks like it's coming back pretty much to its maximum back pretty much to its maximum back pretty much to its maximum capacity. 9.41, 9.44.

capacity. 9.41, 9.44. capacity. 9.41, 9.44. Yeah, that's as much as we're going to Yeah, that's as much as we're going to Yeah, that's as much as we're going to get for a 10 gig network. Network is get for a 10 gig network. Network is get for a 10 gig network. Network is ready, software is ready. Now it's time ready, software is ready. Now it's time ready, software is ready. Now it's time to try this playbook. to try this playbook. to try this playbook. So first we're going to try RPC, which So first we're going to try RPC, which So first we're going to try RPC, which is under advanced and right here, is under advanced and right here, is under advanced and right here, clustering two Ryzen AI Halos with RPC. clustering two Ryzen AI Halos with RPC. clustering two Ryzen AI Halos with RPC. I already went through some of this I already went through some of this I already went through some of this installation here. I did this step, this installation here. I did this step, this installation here. I did this step, this step, this step. And this is interesting step, this step. And this is interesting step, this step. And this is interesting right here. Very, very interesting. AMD right here. Very, very interesting. AMD right here. Very, very interesting. AMD TTM. This lets you find out how much TTM. This lets you find out how much TTM. This lets you find out how much shared memory there is. And I got bitten shared memory there is. And I got bitten shared memory there is. And I got bitten by this on one of the machines. I'll by this on one of the machines. I'll by this on one of the machines. I'll tell you why. If we run this, you'll see tell you why. If we run this, you'll see tell you why. If we run this, you'll see that I have a total system memory of that I have a total system memory of that I have a total system memory of 125. It's 128, but you know, there's 125. It's 128, but you know, there's 125. It's 128, but you know, there's some for the system. And total current some for the system. And total current some for the system. And total current TTM pages limit is 120 GB. This is the TTM pages limit is 120 GB. This is the TTM pages limit is 120 GB. This is the reason why we're using Linux instead of reason why we're using Linux instead of reason why we're using Linux instead of Windows. Windows limits this to 96 as Windows. Windows limits this to 96 as Windows. Windows limits this to 96 as maximum. Linux can go up to 120, but not maximum. Linux can go up to 120, but not maximum. Linux can go up to 120, but not by default. You have to run this command by default. You have to run this command by default. You have to run this command right here. It's part of the playbook in right here. It's part of the playbook in right here. It's part of the playbook in order to give it or allocate 120 GB, order to give it or allocate 120 GB, order to give it or allocate 120 GB, which means the GPU can use up to that which means the GPU can use up to that which means the GPU can use up to that much, allowing you to fit models much, allowing you to fit models much, allowing you to fit models theoretically up to 240 GB in size. Now theoretically up to 240 GB in size. Now theoretically up to 240 GB in size. Now the issue that I ran into was really the issue that I ran into was really the issue that I ran into was really funny because Well, it wasn't that funny because Well, it wasn't that funny because Well, it wasn't that funny.

funny. funny. >> [laughter] >> [laughter] >> [laughter] >> I didn't think so at the time cuz it >> I didn't think so at the time cuz it >> I didn't think so at the time cuz it took me a little while to figure it out. took me a little while to figure it out. took me a little while to figure it out. The Windows version automatically sets The Windows version automatically sets The Windows version automatically sets it to 64 GB. And this is not something it to 64 GB. And this is not something it to 64 GB. And this is not something that you can discover in the BIOS. In that you can discover in the BIOS. In that you can discover in the BIOS. In fact, I went into the BIOS and tried to fact, I went into the BIOS and tried to fact, I went into the BIOS and tried to find that setting like some other Ryzen find that setting like some other Ryzen find that setting like some other Ryzen machines allow you to do, but it wasn't machines allow you to do, but it wasn't machines allow you to do, but it wasn't there. It's not exposed in the BIOS. You there. It's not exposed in the BIOS. You there. It's not exposed in the BIOS. You can only modify that setting using the can only modify that setting using the can only modify that setting using the AMD Ryzen AI developer center. Even AMD Ryzen AI developer center. Even AMD Ryzen AI developer center. Even after I put Linux on that box, I after I put Linux on that box, I after I put Linux on that box, I couldn't change it in the BIOS. I couldn't change it in the BIOS. I couldn't change it in the BIOS. I couldn't change it using this AMD TTM couldn't change it using this AMD TTM couldn't change it using this AMD TTM set 120. In fact, I ran that and it did set 120. In fact, I ran that and it did set 120. In fact, I ran that and it did not work. The system was limited to 64 not work. The system was limited to 64 not work. The system was limited to 64 and llama.cpp RPC was detecting only 64. and llama.cpp RPC was detecting only 64. and llama.cpp RPC was detecting only 64. But, I'm getting ahead of myself. So, in But, I'm getting ahead of myself. So, in But, I'm getting ahead of myself. So, in order to fix that, you need to go into order to fix that, you need to go into order to fix that, you need to go into the settings tab of the AMD Ryzen AI the settings tab of the AMD Ryzen AI the settings tab of the AMD Ryzen AI developer center, which comes with the developer center, which comes with the developer center, which comes with the Linux installation here. And that Linux installation here. And that Linux installation here. And that software allows you to tweak the GPU software allows you to tweak the GPU software allows you to tweak the GPU memory allocation. As far as I know, memory allocation. As far as I know, memory allocation. As far as I know, that's the only way to do it. Now, you that's the only way to do it. Now, you that's the only way to do it. Now, you also need to determine the name of the also need to determine the name of the also need to determine the name of the interface. Unlike the DGX Spark, which interface. Unlike the DGX Spark, which interface. Unlike the DGX Spark, which has a lot of interfaces, this one only has a lot of interfaces, this one only has a lot of interfaces, this one only has one. There it is, right there. And has one. There it is, right there. And has one. There it is, right there. And most likely, yours will also have that most likely, yours will also have that most likely, yours will also have that same name. Having the interface name same name. Having the interface name same name. Having the interface name will allow you to confirm the speed will allow you to confirm the speed will allow you to confirm the speed that's available to you. So, that's 10 that's available to you. So, that's 10 that's available to you. So, that's 10 gigabit right there. So, we're good with gigabit right there. So, we're good with gigabit right there. So, we're good with the connections. Now, this gives you two the connections. Now, this gives you two the connections. Now, this gives you two installation options, lemonade SDK or installation options, lemonade SDK or installation options, lemonade SDK or manual source build. I actually like manual source build. I actually like manual source build. I actually like doing the manual source build. It's not doing the manual source build. It's not doing the manual source build. It's not that hard. You just go to llama.cpp's that hard. You just go to llama.cpp's that hard. You just go to llama.cpp's GitHub and it's good to poke around in GitHub and it's good to poke around in GitHub and it's good to poke around in here. What's good about this project is here. What's good about this project is here. What's good about this project is that it's so popular that it kind of had that it's so popular that it kind of had that it's so popular that it kind of had to be really well documented. And if to be really well documented. And if to be really well documented. And if you're doing any sort of AI work on your you're doing any sort of AI work on your you're doing any sort of AI work on your machines, you should be familiar with

machines, you should be familiar with machines, you should be familiar with these steps. Just go on the GitHub's and these steps. Just go on the GitHub's and these steps. Just go on the GitHub's and use the quick start there and do the use the quick start there and do the use the quick start there and do the build. Do it right now. build. Do it right now. build. Do it right now. >> [laughter] >> [laughter] >> [laughter] >> Then, you can just keep going through >> Then, you can just keep going through >> Then, you can just keep going through the playbook. I don't want to bore you the playbook. I don't want to bore you the playbook. I don't want to bore you with all the steps to go through this with all the steps to go through this with all the steps to go through this because it's pretty much verbatim. You because it's pretty much verbatim. You because it's pretty much verbatim. You just go through these steps one by one. just go through these steps one by one. just go through these steps one by one. I did that and I want to share the I did that and I want to share the I did that and I want to share the results. One more thing to notice here results. One more thing to notice here results. One more thing to notice here is the playbook uses GLM 4.7. See, I is the playbook uses GLM 4.7. See, I is the playbook uses GLM 4.7. See, I told you it was verbatim. I'm reading it told you it was verbatim. I'm reading it told you it was verbatim. I'm reading it right here. It's a 358 billion parameter right here. It's a 358 billion parameter right here. It's a 358 billion parameter model in the Q4 KX quantization from model in the Q4 KX quantization from model in the Q4 KX quantization from Onslaught. Onslaught. Onslaught. That's actually not what it is because That's actually not what it is because That's actually not what it is because here, they tell you to include the UDQ4 here, they tell you to include the UDQ4 here, they tell you to include the UDQ4 KX. And that's what I use, the UD KX. And that's what I use, the UD KX. And that's what I use, the UD version, but the Q4 KXL and Q4 KM will version, but the Q4 KXL and Q4 KM will version, but the Q4 KXL and Q4 KM will also work. The KM is what I usually use, also work. The KM is what I usually use, also work. The KM is what I usually use, so I was wondering, why do they have so I was wondering, why do they have so I was wondering, why do they have UDQ4? Well, this model is on Sloth's UDQ4? Well, this model is on Sloth's UDQ4? Well, this model is on Sloth's GGUF, good guff. And the difference with GGUF, good guff. And the difference with GGUF, good guff. And the difference with UD, that just means on Sloth dynamic. UD, that just means on Sloth dynamic. UD, that just means on Sloth dynamic. So, if I back up here, under GLM 4.7 So, if I back up here, under GLM 4.7 So, if I back up here, under GLM 4.7 GGUF, you have all these different GGUF, you have all these different GGUF, you have all these different quantizations that on Sloth was nice quantizations that on Sloth was nice quantizations that on Sloth was nice enough to quantize for you. So, all you enough to quantize for you. So, all you enough to quantize for you. So, all you have to do is just download them.

have to do is just download them. have to do is just download them. Instead of Q4KM, Q4KS, we're getting Instead of Q4KM, Q4KS, we're getting Instead of Q4KM, Q4KS, we're getting this UGQ4KXL. this UGQ4KXL. this UGQ4KXL. Unlike the other ones where it's uniform Unlike the other ones where it's uniform Unlike the other ones where it's uniform 4-bit quantization, like the classic 4-bit quantization, like the classic 4-bit quantization, like the classic Q4KM, this one spends more bits on the Q4KM, this one spends more bits on the Q4KM, this one spends more bits on the tensors that matter. Attention, the tensors that matter. Attention, the tensors that matter. Attention, the router embeddings, and fewer bits on the router embeddings, and fewer bits on the router embeddings, and fewer bits on the huge, rarely hit mixture of experts huge, rarely hit mixture of experts huge, rarely hit mixture of experts layers. And that's actually a good layers. And that's actually a good layers. And that's actually a good thing. For a 358 billion mixture of thing. For a 358 billion mixture of thing. For a 358 billion mixture of experts model like this one, it's experts model like this one, it's experts model like this one, it's supposed to be better and have better supposed to be better and have better supposed to be better and have better payoff for the quality per byte. Yet, payoff for the quality per byte. Yet, payoff for the quality per byte. Yet, it's exactly the same size. I've talked it's exactly the same size. I've talked it's exactly the same size. I've talked about this on the channel before. And about this on the channel before. And about this on the channel before. And Bartowski is another quantizer, pretty Bartowski is another quantizer, pretty Bartowski is another quantizer, pretty famous on the internet. They all use famous on the internet. They all use famous on the internet. They all use like little techniques to tweak the like little techniques to tweak the like little techniques to tweak the quantizations to get more juice out of quantizations to get more juice out of quantizations to get more juice out of it. It's not just about running a bigger it. It's not just about running a bigger it. It's not just about running a bigger model. It's also running a higher model. It's also running a higher model. It's also running a higher quality quantization of a big model. quality quantization of a big model. quality quantization of a big model. But, neither one of these quants will But, neither one of these quants will But, neither one of these quants will fit on a single 128 GB box. The second fit on a single 128 GB box. The second fit on a single 128 GB box. The second Halo is what lets you serve a good quant Halo is what lets you serve a good quant Halo is what lets you serve a good quant instead of dropping down, I don't know, instead of dropping down, I don't know, instead of dropping down, I don't know, one of these QIQ2Ms one of these QIQ2Ms one of these QIQ2Ms or IQ1S or IQ1S or IQ1S quants, which are probably going to mess quants, which are probably going to mess quants, which are probably going to mess up your output. I made a whole separate up your output. I made a whole separate up your output. I made a whole separate video on the latters, going from video on the latters, going from video on the latters, going from different quantizations and what kind of different quantizations and what kind of different quantizations and what kind of output they produce. I'll link to that output they produce. I'll link to that output they produce. I'll link to that down below. I'm running it now, and with down below. I'm running it now, and with down below. I'm running it now, and with either one of these approaches, it takes either one of these approaches, it takes either one of these approaches, it takes a little while for the system to start a little while for the system to start a little while for the system to start up. But, it's a lot easier with the RPC up. But, it's a lot easier with the RPC up. But, it's a lot easier with the RPC server, the way we're doing it right server, the way we're doing it right server, the way we're doing it right now. This is basically llama.cpp now. This is basically llama.cpp now. This is basically llama.cpp borrowing the second Halo's memory, and borrowing the second Halo's memory, and borrowing the second Halo's memory, and it's a much simpler approach. Halo 2

it's a much simpler approach. Halo 2 it's a much simpler approach. Halo 2 simply exposes its GPU as an RPC worker, simply exposes its GPU as an RPC worker, simply exposes its GPU as an RPC worker, and Halo 1 is like the root node that and Halo 1 is like the root node that and Halo 1 is like the root node that points to it and uses it. And it's up. points to it and uses it. And it's up. points to it and uses it. And it's up. It's doing the pre-fill, and there it It's doing the pre-fill, and there it It's doing the pre-fill, and there it goes. It is generating the text. We're goes. It is generating the text. We're goes. It is generating the text. We're getting about getting about getting about uh 8.2 tokens per second here. Remember, uh 8.2 tokens per second here. Remember, uh 8.2 tokens per second here. Remember, folks, this is GLM 4.7. It's pretty big. folks, this is GLM 4.7. It's pretty big. folks, this is GLM 4.7. It's pretty big. I'm going to run Rock-SMI I'm going to run Rock-SMI I'm going to run Rock-SMI here, and wow, interesting. Okay, so here, and wow, interesting. Okay, so here, and wow, interesting. Okay, so VRAM is at 35% and the power is only at VRAM is at 35% and the power is only at VRAM is at 35% and the power is only at about 43 W. GPU is only about halfway about 43 W. GPU is only about halfway about 43 W. GPU is only about halfway utilized. This is only on one machine, utilized. This is only on one machine, utilized. This is only on one machine, by the way, so you'd imagine that it's by the way, so you'd imagine that it's by the way, so you'd imagine that it's the same thing is happening on the the same thing is happening on the the same thing is happening on the second machine. Oh, now it's going. second machine. Oh, now it's going. second machine. Oh, now it's going. Okay, so the GPU is back up to 100%. Okay, so the GPU is back up to 100%. Okay, so the GPU is back up to 100%. That's good. It's kind of bouncing That's good. It's kind of bouncing That's good. It's kind of bouncing around, probably between the decode around, probably between the decode around, probably between the decode stage and the uh processing stage. All stage and the uh processing stage. All stage and the uh processing stage. All right, it's almost done. By the way, I I right, it's almost done. By the way, I I right, it's almost done. By the way, I I I got to take a moment to say this. Uh I got to take a moment to say this. Uh I got to take a moment to say this. Uh you can't complain in the comments below you can't complain in the comments below you can't complain in the comments below that I don't ever do big models. Hey, that I don't ever do big models. Hey, that I don't ever do big models. Hey, big model. Another big model coming up big model. Another big model coming up big model. Another big model coming up shortly here, an even bigger model. Oh, shortly here, an even bigger model. Oh, shortly here, an even bigger model. Oh, here we go. 7.93, here we go. 7.93, here we go. 7.93, 7.69, 7.69, 7.69, 7.6.

7.6. 7.6. Within that range for the concurrency of Within that range for the concurrency of Within that range for the concurrency of one, by the way. This is like basically one, by the way. This is like basically one, by the way. This is like basically chatting with this thing, and that's how chatting with this thing, and that's how chatting with this thing, and that's how many tokens per second we're getting out many tokens per second we're getting out many tokens per second we're getting out of this. Now, if we take a look at of this. Now, if we take a look at of this. Now, if we take a look at concurrency four, which I also ran, concurrency four, which I also ran, concurrency four, which I also ran, we're almost doubling. That's not bad, we're almost doubling. That's not bad, we're almost doubling. That's not bad, actually. But, it only goes up to a actually. But, it only goes up to a actually. But, it only goes up to a certain point with llama.cpp. With certain point with llama.cpp. With certain point with llama.cpp. With concurrency of four for TG 128, we're concurrency of four for TG 128, we're concurrency of four for TG 128, we're getting 13, and for TG 256, we're getting 13, and for TG 256, we're getting 13, and for TG 256, we're getting 13.5. But then, if you raise getting 13.5. But then, if you raise getting 13.5. But then, if you raise your prompt up a little bit, a little your prompt up a little bit, a little your prompt up a little bit, a little higher to 2048, our concurrency of four higher to 2048, our concurrency of four higher to 2048, our concurrency of four falls back to 5.5 tokens per second. falls back to 5.5 tokens per second. falls back to 5.5 tokens per second. This is where this workflow breaks down, This is where this workflow breaks down, This is where this workflow breaks down, and with llama.cpp, you need a slightly and with llama.cpp, you need a slightly and with llama.cpp, you need a slightly different approach. I made a whole different approach. I made a whole different approach. I made a whole different video about that, how to different video about that, how to different video about that, how to handle llama.cpp with lots of handle llama.cpp with lots of handle llama.cpp with lots of concurrencies. Why do you need that? concurrencies. Why do you need that? concurrencies. Why do you need that? Well, if you're doing things like Well, if you're doing things like Well, if you're doing things like multiple requests at the same time with multiple requests at the same time with multiple requests at the same time with that kind of setup, if you're in a that kind of setup, if you're in a that kind of setup, if you're in a workspace with multiple people using it, workspace with multiple people using it, workspace with multiple people using it, or if you have an agent, and if you're or if you have an agent, and if you're or if you have an agent, and if you're designing some kind of multi-agent designing some kind of multi-agent designing some kind of multi-agent architecture, then that matters. And architecture, then that matters. And architecture, then that matters. And I'll link to that video down below. But I'll link to that video down below. But I'll link to that video down below. But here we have a different approach to here we have a different approach to here we have a different approach to handling concurrency, and that's where handling concurrency, and that's where handling concurrency, and that's where the other workflow comes in. And that the other workflow comes in. And that the other workflow comes in. And that one is a little bit more difficult. And one is a little bit more difficult. And one is a little bit more difficult. And you say, "Well, why do I want a more you say, "Well, why do I want a more you say, "Well, why do I want a more difficult one, right? When this works difficult one, right? When this works difficult one, right? When this works perfectly fine." And if it works for perfectly fine." And if it works for perfectly fine." And if it works for your use case, go ahead and use it. RPC your use case, go ahead and use it. RPC your use case, go ahead and use it. RPC still lets you fit a giant model and run still lets you fit a giant model and run still lets you fit a giant model and run one conversation at a time, maybe two.

one conversation at a time, maybe two. one conversation at a time, maybe two. But VLLM with vLLM, it's a different But VLLM with vLLM, it's a different But VLLM with vLLM, it's a different story. story. story. >> [music] >> [music] >> [music] >> Your Ryzen AI Halo is already capable of >> Your Ryzen AI Halo is already capable of >> Your Ryzen AI Halo is already capable of running large language models locally, running large language models locally, running large language models locally, but clustering takes it a further by but clustering takes it a further by but clustering takes it a further by combining GPU memory. You already know combining GPU memory. You already know combining GPU memory. You already know all this from watching the rest of my all this from watching the rest of my all this from watching the rest of my video, but in case you skipped around, video, but in case you skipped around, video, but in case you skipped around, this is a this is a little refresher. this is a this is a little refresher. this is a this is a little refresher. But this playbook teaches you how to But this playbook teaches you how to But this playbook teaches you how to cluster two of them using vLLM, ROCm cluster two of them using vLLM, ROCm cluster two of them using vLLM, ROCm communication collectives library. This communication collectives library. This communication collectives library. This It's an acronym within an acronym. With It's an acronym within an acronym. With It's an acronym within an acronym. With vLLM, and we're going to run Qwen 3.5 vLLM, and we're going to run Qwen 3.5 vLLM, and we're going to run Qwen 3.5 397B, which is a much bigger model than 397B, which is a much bigger model than 397B, which is a much bigger model than the previous one. Most of this the previous one. Most of this the previous one. Most of this documentation is the same as the one we documentation is the same as the one we documentation is the same as the one we just went through. You're getting the just went through. You're getting the just went through. You're getting the network link speeds, which we're fine network link speeds, which we're fine network link speeds, which we're fine with. You're installing a bunch of with. You're installing a bunch of with. You're installing a bunch of dependencies, setting your memory to the dependencies, setting your memory to the dependencies, setting your memory to the highest possible. Here's where things highest possible. Here's where things highest possible. Here's where things get different, vLLM containers. And this get different, vLLM containers. And this get different, vLLM containers. And this is going to use a Podman container and is going to use a Podman container and is going to use a Podman container and tells you exactly what to run to launch tells you exactly what to run to launch tells you exactly what to run to launch it. Now, this is going to take a while. it. Now, this is going to take a while. it. Now, this is going to take a while. In my case, for this large model, it In my case, for this large model, it In my case, for this large model, it took probably 15 minutes to actually took probably 15 minutes to actually took probably 15 minutes to actually launch it. It uses another technology launch it. It uses another technology launch it. It uses another technology called Ray, which is responsible for called Ray, which is responsible for called Ray, which is responsible for doing the multi-node communication.

doing the multi-node communication. doing the multi-node communication. Yeah, there's a lot of stuff going on Yeah, there's a lot of stuff going on Yeah, there's a lot of stuff going on here. Let's see the results. Oh, on here. Let's see the results. Oh, on here. Let's see the results. Oh, on machine one you start Ray, on machine machine one you start Ray, on machine machine one you start Ray, on machine two you you the cluster, and then you two you you the cluster, and then you two you you the cluster, and then you serve the model using this right here. serve the model using this right here. serve the model using this right here. If you get one flag wrong, you have to If you get one flag wrong, you have to If you get one flag wrong, you have to start over and wait another 15 minutes. start over and wait another 15 minutes. start over and wait another 15 minutes. Don't ask me how I know. And this is why Don't ask me how I know. And this is why Don't ask me how I know. And this is why I use smaller models to try it with. I use smaller models to try it with. I use smaller models to try it with. Don't start with the large models first. Don't start with the large models first. Don't start with the large models first. Use a model with similar architecture. Use a model with similar architecture. Use a model with similar architecture. In this case, it'll be mixture of In this case, it'll be mixture of In this case, it'll be mixture of experts Qwen 3.5, but a smaller one. All experts Qwen 3.5, but a smaller one. All experts Qwen 3.5, but a smaller one. All right, so here we go. Llama Benchy, I'm right, so here we go. Llama Benchy, I'm right, so here we go. Llama Benchy, I'm going to point it to this machine, and going to point it to this machine, and going to point it to this machine, and we're going to run a couple of PPs, we're going to run a couple of PPs, we're going to run a couple of PPs, prompt processing 512 and 2048, and a prompt processing 512 and 2048, and a prompt processing 512 and 2048, and a couple of TGs, token generation 128 and couple of TGs, token generation 128 and couple of TGs, token generation 128 and 256 for concurrency one and concurrency 256 for concurrency one and concurrency 256 for concurrency one and concurrency four. Here we go. This is the big one. four. Here we go. This is the big one. four. Here we go. This is the big one. Qwen 3.5 397B, and it's going. I can Qwen 3.5 397B, and it's going. I can Qwen 3.5 397B, and it's going. I can actually hear both machines. It's not actually hear both machines. It's not actually hear both machines. It's not breaking. It's going. You can also run breaking. It's going. You can also run breaking. It's going. You can also run rocm-smi on each one of these machines. rocm-smi on each one of these machines. rocm-smi on each one of these machines. Here it is, showing that 36% of VRAM's Here it is, showing that 36% of VRAM's Here it is, showing that 36% of VRAM's being used and 100% of the GPU. This being used and 100% of the GPU. This being used and 100% of the GPU. This also monitors the temperature for you also monitors the temperature for you also monitors the temperature for you and the power output. So, we're using 60 and the power output. So, we're using 60 and the power output. So, we're using 60 W, 52° C. It's a full amount of GPU W, 52° C. It's a full amount of GPU W, 52° C. It's a full amount of GPU usage, but not a crazy amount of VRAM usage, but not a crazy amount of VRAM usage, but not a crazy amount of VRAM usage. This is actually tensor usage. This is actually tensor usage. This is actually tensor parallelism at work here. TP = 2. And parallelism at work here. TP = 2. And parallelism at work here. TP = 2. And we're using about 110 GB per box here we're using about 110 GB per box here we're using about 110 GB per box here for total of 220 pulled. Can you hear for total of 220 pulled. Can you hear for total of 220 pulled. Can you hear the machines?

the machines? the machines? >> [laughter] >> [laughter] >> [laughter] >> I can hear it. Rickel talking over the >> I can hear it. Rickel talking over the >> I can hear it. Rickel talking over the 10 GB link at about 7.81 10 GB link at about 7.81 10 GB link at about 7.81 tokens per second and up to almost 18 tokens per second and up to almost 18 tokens per second and up to almost 18 tokens per second for concurrency of tokens per second for concurrency of tokens per second for concurrency of four. That's what I'm talking about. So, four. That's what I'm talking about. So, four. That's what I'm talking about. So, if you remember group three from my if you remember group three from my if you remember group three from my previous video, the one that I kind of previous video, the one that I kind of previous video, the one that I kind of set up, well, this is for those of you set up, well, this is for those of you set up, well, this is for those of you that are in group three. If you're going that are in group three. If you're going that are in group three. If you're going to be setting this up to do work, you to be setting this up to do work, you to be setting this up to do work, you have a couple of different approaches, have a couple of different approaches, have a couple of different approaches, and if you're going to be clustering and if you're going to be clustering and if you're going to be clustering multiple machines, then AMD gives you a multiple machines, then AMD gives you a multiple machines, then AMD gives you a couple of different options to do that. couple of different options to do that. couple of different options to do that. Llama RPC is simple. Rickel is a little Llama RPC is simple. Rickel is a little Llama RPC is simple. Rickel is a little bit more complicated, but it scales a bit more complicated, but it scales a bit more complicated, but it scales a little bit better. And it's like running little bit better. And it's like running little bit better. And it's like running a little mini data center at home or in a little mini data center at home or in a little mini data center at home or in your office. This is the technology your office. This is the technology your office. This is the technology stack that you'll use if you transition stack that you'll use if you transition stack that you'll use if you transition into bigger AMD infrastructure. Now, AMD into bigger AMD infrastructure. Now, AMD into bigger AMD infrastructure. Now, AMD also alludes to having four node also alludes to having four node also alludes to having four node clusters as well. And if you want to see clusters as well. And if you want to see clusters as well. And if you want to see my attempts at another machines in the my attempts at another machines in the my attempts at another machines in the four node cluster with this very chip, four node cluster with this very chip, four node cluster with this very chip, watch this video here. Thanks for watch this video here. Thanks for watch this video here. Thanks for watching and I'll see you next time. watching and I'll see you next time. watching and I'll see you next time. >> [music]