I Ran A 27B Model On A Hand-Sized PC… Didn't Expect This

This thing fits in one hand. And I This thing fits in one hand. And I wanted to know if it can run a serious wanted to know if it can run a serious wanted to know if it can run a serious AI model, not a toy. All right, a 27 AI model, not a toy. All right, a 27 AI model, not a toy. All right, a 27 billion parameter model. These are billion parameter model. These are billion parameter model. These are getting really good now. You know, the getting really good now. You know, the getting really good now. You know, the kind of model that you normally throw a kind of model that you normally throw a kind of model that you normally throw a big GPU at. And we'll do that. But big GPU at. And we'll do that. But big GPU at. And we'll do that. But first, we'll start off with just this. first, we'll start off with just this. first, we'll start off with just this. And guess what? It can. But that's not And guess what? It can. But that's not And guess what? It can. But that's not the surprising part. So, I first saw the surprising part. So, I first saw the surprising part. So, I first saw this at CES this year. This is the Kadas this at CES this year. This is the Kadas this at CES this year. This is the Kadas Mind Pro. It is literally the smallest Mind Pro. It is literally the smallest Mind Pro. It is literally the smallest Core Ultra X7 358H machine that I've Core Ultra X7 358H machine that I've Core Ultra X7 358H machine that I've seen. Panther Lake by Intel. This little seen. Panther Lake by Intel. This little seen. Panther Lake by Intel. This little thing has 64 gigs of memory on it. And thing has 64 gigs of memory on it. And thing has 64 gigs of memory on it. And Intel's Panther Lake, of course, Intel's Panther Lake, of course, Intel's Panther Lake, of course, includes the ARC B390 graphics baked includes the ARC B390 graphics baked includes the ARC B390 graphics baked right in. That's the iGPU or integrated right in. That's the iGPU or integrated right in. That's the iGPU or integrated GPU. And this is really cool because you GPU. And this is really cool because you GPU. And this is really cool because you can get a dock for it. A dock that fits can get a dock for it. A dock that fits can get a dock for it. A dock that fits it perfectly with this little connector. it perfectly with this little connector. it perfectly with this little connector. It's a proprietary connector. And this It's a proprietary connector. And this It's a proprietary connector. And this is not just a dock. It actually has a is not just a dock. It actually has a is not just a dock. It actually has a discrete GPU right inside. And with that discrete GPU right inside. And with that discrete GPU right inside. And with that ultra fast connection, well, the ultra fast connection, well, the ultra fast connection, well, the communication between them is pretty communication between them is pretty communication between them is pretty fast. You can also get it with a smaller fast. You can also get it with a smaller fast. You can also get it with a smaller dock, which also sits in there perfectly dock, which also sits in there perfectly dock, which also sits in there perfectly magnetized. And this provides other magnetized. And this provides other magnetized. And this provides other connections for you. And all these connections for you. And all these connections for you. And all these things can run your models. Well, not things can run your models. Well, not things can run your models. Well, not the dock. This dock can. And this thing the dock. This dock can. And this thing the dock. This dock can. And this thing can. So, I'm going to be throwing Quen can. So, I'm going to be throwing Quen can. So, I'm going to be throwing Quen 3.8 at it. That's the 27B version. a 3.8 at it. That's the 27B version. a 3.8 at it. That's the 27B version. a very very popular and capable model and very very popular and capable model and very very popular and capable model and it's about 18 gigabytes of weights at it's about 18 gigabytes of weights at it's about 18 gigabytes of weights at Q4KM quantization. So that's a 4bit Q4KM quantization. So that's a 4bit Q4KM quantization. So that's a 4bit model rounded down from 16 bits per

model rounded down from 16 bits per model rounded down from 16 bits per number. K means that it's the newer number. K means that it's the newer number. K means that it's the newer scheme that keeps the sensitive part of scheme that keeps the sensitive part of scheme that keeps the sensitive part of the model at higher precision and M the model at higher precision and M the model at higher precision and M means medium or how much it protects. means medium or how much it protects. means medium or how much it protects. But my question isn't does it run. Of But my question isn't does it run. Of But my question isn't does it run. Of course it does. Everything runs if course it does. Everything runs if course it does. Everything runs if you're patient enough, right? you're patient enough, right? you're patient enough, right? It's how fast I can make it go. Because It's how fast I can make it go. Because It's how fast I can make it go. Because a model of this size that's actually a model of this size that's actually a model of this size that's actually good at code. And if it's typing at four good at code. And if it's typing at four good at code. And if it's typing at four tokens per second, then you're going to tokens per second, then you're going to tokens per second, then you're going to give up on it pretty quickly and be back give up on it pretty quickly and be back give up on it pretty quickly and be back on the cloud API. So before I bolt on the cloud API. So before I bolt on the cloud API. So before I bolt anything onto this thing, what can it anything onto this thing, what can it anything onto this thing, what can it already do just on this tiny little box? already do just on this tiny little box? already do just on this tiny little box? Well, Intel gives you three separate Well, Intel gives you three separate Well, Intel gives you three separate ways to talk to the same integrated GPU. ways to talk to the same integrated GPU. ways to talk to the same integrated GPU. Vulcan, Sickle, and Open Veno. Plus, you Vulcan, Sickle, and Open Veno. Plus, you Vulcan, Sickle, and Open Veno. Plus, you also have the plain old CPU. So, I also have the plain old CPU. So, I also have the plain old CPU. So, I measured all four. We're going to be measured all four. We're going to be measured all four. We're going to be talking about two numbers here. One is talking about two numbers here. One is talking about two numbers here. One is the prefill phase or the reading and the the prefill phase or the reading and the the prefill phase or the reading and the other one is the decode phase which is other one is the decode phase which is other one is the decode phase which is the writing that's the token generation the writing that's the token generation the writing that's the token generation phase. So prompt processing on the CPU phase. So prompt processing on the CPU phase. So prompt processing on the CPU is about 17 tokens a second is about 17 tokens a second is about 17 tokens a second and we jump all the way up to 446 tokens and we jump all the way up to 446 tokens and we jump all the way up to 446 tokens per second using open veno. 27 times big per second using open veno. 27 times big per second using open veno. 27 times big difference there. Now for writing we difference there. Now for writing we difference there. Now for writing we flip a little bit under five tokens per flip a little bit under five tokens per flip a little bit under five tokens per second on the CPU. 5.4 4 on Vulcan, 5.7 second on the CPU. 5.4 4 on Vulcan, 5.7 second on the CPU. 5.4 4 on Vulcan, 5.7 on sickle, 2.6 on Open Veno. That's the on sickle, 2.6 on Open Veno. That's the on sickle, 2.6 on Open Veno. That's the whole range. But don't worry, we're whole range. But don't worry, we're whole range. But don't worry, we're going to squeeze a little more out of going to squeeze a little more out of going to squeeze a little more out of this lemon in just a bit. The bottleneck this lemon in just a bit. The bottleneck this lemon in just a bit. The bottleneck here is memory. Capacity is the size of here is memory. Capacity is the size of here is memory. Capacity is the size of the tank. And bandwidth is the width of

the tank. And bandwidth is the width of the tank. And bandwidth is the width of the pipe, how fast you get it back out. the pipe, how fast you get it back out. the pipe, how fast you get it back out. For every single token that this thing For every single token that this thing For every single token that this thing writes means streaming all 18 gigs of writes means streaming all 18 gigs of writes means streaming all 18 gigs of the weights past the chip, and that's the weights past the chip, and that's the weights past the chip, and that's once for every single word. And on once for every single word. And on once for every single word. And on integrated graphics, the GPU and CPU integrated graphics, the GPU and CPU integrated graphics, the GPU and CPU share one pool of system memory. So the share one pool of system memory. So the share one pool of system memory. So the GPU is just sitting there waiting on GPU is just sitting there waiting on GPU is just sitting there waiting on memory. By the way, here's how I know memory. By the way, here's how I know memory. By the way, here's how I know the bandwidth. Model size is 17.66 the bandwidth. Model size is 17.66 the bandwidth. Model size is 17.66 times the token rate, which is 5.42. So times the token rate, which is 5.42. So times the token rate, which is 5.42. So it's about 100 GB a second. That's not it's about 100 GB a second. That's not it's about 100 GB a second. That's not the advertised. That's the actual the advertised. That's the actual the advertised. That's the actual measured real bandwidth of this machine. measured real bandwidth of this machine. measured real bandwidth of this machine. And no back end is going to just And no back end is going to just And no back end is going to just automatically magically fix this. The automatically magically fix this. The automatically magically fix this. The only way through is to stop producing only way through is to stop producing only way through is to stop producing one word per pass. It's called one word per pass. It's called one word per pass. It's called speculative decoding. And I've made speculative decoding. And I've made speculative decoding. And I've made videos about this before. Basically, videos about this before. Basically, videos about this before. Basically, this involves a smaller, faster model. this involves a smaller, faster model. this involves a smaller, faster model. Let's call it a draft model. That's Let's call it a draft model. That's Let's call it a draft model. That's actually what it's called. It guesses actually what it's called. It guesses actually what it's called. It guesses several tokens ahead. And the big model several tokens ahead. And the big model several tokens ahead. And the big model checks all those guesses in one pass. checks all those guesses in one pass. checks all those guesses in one pass. And a pass cost the same whether it's And a pass cost the same whether it's And a pass cost the same whether it's checking one token or five. So, every checking one token or five. So, every checking one token or five. So, every guest that gets accepted is basically guest that gets accepted is basically guest that gets accepted is basically free. Some of the quen quans that you'll free. Some of the quen quans that you'll free. Some of the quen quans that you'll find on Hugging Face, for example, will find on Hugging Face, for example, will find on Hugging Face, for example, will ship this right next to the model. just ship this right next to the model. just ship this right next to the model. just an extra 1 and 1/2 gigs or so. That's an extra 1 and 1/2 gigs or so. That's an extra 1 and 1/2 gigs or so. That's basically a rounding error. So, you basically a rounding error. So, you basically a rounding error. So, you should be using that if you're not should be using that if you're not should be using that if you're not already. On the integrated GPU, the already. On the integrated GPU, the already. On the integrated GPU, the iGPU, it went from about five tokens a iGPU, it went from about five tokens a iGPU, it went from about five tokens a second here to 11. That's more than second here to 11. That's more than second here to 11. That's more than double and it's free. But if you notice, double and it's free. But if you notice, double and it's free. But if you notice, the shape of this chart is interesting.

the shape of this chart is interesting. the shape of this chart is interesting. I swept the draft depth, or in other I swept the draft depth, or in other I swept the draft depth, or in other words, how many tokens ahead it allows words, how many tokens ahead it allows words, how many tokens ahead it allows you to guess. And uh a little spoiler you to guess. And uh a little spoiler you to guess. And uh a little spoiler for you here, okay, for the future, I for you here, okay, for the future, I for you here, okay, for the future, I also did it on our 5060 Ti that's inside also did it on our 5060 Ti that's inside also did it on our 5060 Ti that's inside here. And both devices actually peaked here. And both devices actually peaked here. And both devices actually peaked at Nmax 3. That's how many tokens ahead at Nmax 3. That's how many tokens ahead at Nmax 3. That's how many tokens ahead you're peaking even though one of them you're peaking even though one of them you're peaking even though one of them is five times faster than the other. I is five times faster than the other. I is five times faster than the other. I think you know which one is which, but think you know which one is which, but think you know which one is which, but we'll be back to that in a moment. we'll be back to that in a moment. we'll be back to that in a moment. Because past a certain point, you're Because past a certain point, you're Because past a certain point, you're paying for guesses that are basically paying for guesses that are basically paying for guesses that are basically just thrown away. Push it further and it just thrown away. Push it further and it just thrown away. Push it further and it gets pretty ugly. I did an early run and gets pretty ugly. I did an early run and gets pretty ugly. I did an early run and I put N max eight. Um, I put N max eight. Um, I put N max eight. Um, that got me about three and a half that got me about three and a half that got me about three and a half tokens per second. Yeah, slower than not tokens per second. Yeah, slower than not tokens per second. Yeah, slower than not using speculative decoding at all. So, using speculative decoding at all. So, using speculative decoding at all. So, be careful with that flag. By the way, be careful with that flag. By the way, be careful with that flag. By the way, the default for Llama CPP is already the default for Llama CPP is already the default for Llama CPP is already Nmax 3. So, if you turn the draft model Nmax 3. So, if you turn the draft model Nmax 3. So, if you turn the draft model on, leave the depth alone. Don't be on, leave the depth alone. Don't be on, leave the depth alone. Don't be clever. I was clever. It didn't work out clever. I was clever. It didn't work out clever. I was clever. It didn't work out so well for me, did it? So now you have so well for me, did it? So now you have so well for me, did it? So now you have two tricks up your sleeve, but you're two tricks up your sleeve, but you're two tricks up your sleeve, but you're still kind of uh stuck behind the shared still kind of uh stuck behind the shared still kind of uh stuck behind the shared memory bandwidth. What we need is a GPU memory bandwidth. What we need is a GPU memory bandwidth. What we need is a GPU with a memory of its own. I don't expect with a memory of its own. I don't expect with a memory of its own. I don't expect one AI prompt to build an entire project one AI prompt to build an entire project one AI prompt to build an entire project for me. In reality, I'm constantly for me. In reality, I'm constantly for me. In reality, I'm constantly moving between models depending on the moving between models depending on the moving between models depending on the job. GPT for research, Claude for job. GPT for research, Claude for job. GPT for research, Claude for coding, Gemini for massive context, coding, Gemini for massive context, coding, Gemini for massive context, nanobanana, midjourney, flux for images, nanobanana, midjourney, flux for images, nanobanana, midjourney, flux for images, and then I've got seed dance and cling and then I've got seed dance and cling and then I've got seed dance and cling for video. That's why chat lm by Abacus

for video. That's why chat lm by Abacus for video. That's why chat lm by Abacus AI makes sense. It brings day one AI makes sense. It brings day one AI makes sense. It brings day one support for the latest GPT, Claw, support for the latest GPT, Claw, support for the latest GPT, Claw, Gemini, Grock, Deepseek, and more in one Gemini, Grock, Deepseek, and more in one Gemini, Grock, Deepseek, and more in one place the moment they drop. Pick any place the moment they drop. Pick any place the moment they drop. Pick any model from the interface or let route model from the interface or let route model from the interface or let route LLM automatically choose the best model LLM automatically choose the best model LLM automatically choose the best model for each prompt. Create professional for each prompt. Create professional for each prompt. Create professional presentations with graphs and charts and presentations with graphs and charts and presentations with graphs and charts and deep research detailed content. Need deep research detailed content. Need deep research detailed content. Need human sounding copy? Humanize rewrites human sounding copy? Humanize rewrites human sounding copy? Humanize rewrites text to defeat AI detectors. Need text to defeat AI detectors. Need text to defeat AI detectors. Need visuals? Pick frontier or open-source visuals? Pick frontier or open-source visuals? Pick frontier or open-source models. And when you need more than models. And when you need more than models. And when you need more than chat, Abacus AI agent can help build chat, Abacus AI agent can help build chat, Abacus AI agent can help build complex apps and websites, connect complex apps and websites, connect complex apps and websites, connect payments, or run 247 agents that keep payments, or run 247 agents that keep payments, or run 247 agents that keep working through longer tasks. The best working through longer tasks. The best working through longer tasks. The best part is app hosting, back-end database, part is app hosting, back-end database, part is app hosting, back-end database, and O support comes with the and O support comes with the and O support comes with the subscription. All that starts at just subscription. All that starts at just subscription. All that starts at just $10 a month. Way cheaper than paying for $10 a month. Way cheaper than paying for $10 a month. Way cheaper than paying for all those subscriptions separately. all those subscriptions separately. all those subscriptions separately. Check out chatlm.abacus.ai Check out chatlm.abacus.ai Check out chatlm.abacus.ai or click the link below. So, this is or click the link below. So, this is or click the link below. So, this is where this doc comes in handy. Boom. where this doc comes in handy. Boom. where this doc comes in handy. Boom. Sits there on a magnet. Check it out.

Sits there on a magnet. Check it out. Sits there on a magnet. Check it out. Yeah, 16 gigs of VRAM here. And that's Yeah, 16 gigs of VRAM here. And that's Yeah, 16 gigs of VRAM here. And that's the 5060 Ti. The memory belongs to the the 5060 Ti. The memory belongs to the the 5060 Ti. The memory belongs to the GPU and nobody can mess with it. But um GPU and nobody can mess with it. But um GPU and nobody can mess with it. But um when I loaded it up about 2 seconds when I loaded it up about 2 seconds when I loaded it up about 2 seconds later, I hit a pretty awkward fact that later, I hit a pretty awkward fact that later, I hit a pretty awkward fact that uh like an idiot, I kind of forgot about uh like an idiot, I kind of forgot about uh like an idiot, I kind of forgot about the model is 17.6 GB. the model is 17.6 GB. the model is 17.6 GB. Yeah, notice a problem. It doesn't fit Yeah, notice a problem. It doesn't fit Yeah, notice a problem. It doesn't fit by 1.7 GB. Now, here's why this not the by 1.7 GB. Now, here's why this not the by 1.7 GB. Now, here's why this not the end of this. This model isn't one big end of this. This model isn't one big end of this. This model isn't one big lump. It's actually a stack of about 62 lump. It's actually a stack of about 62 lump. It's actually a stack of about 62 layers. M cake. And all these layers layers. M cake. And all these layers layers. M cake. And all these layers don't have to live in the same place. My don't have to live in the same place. My don't have to live in the same place. My stomach llama CPP has a flag for that stomach llama CPP has a flag for that stomach llama CPP has a flag for that actually. It's called NGL. You might actually. It's called NGL. You might actually. It's called NGL. You might have seen it. NGL is short for NGPU have seen it. NGL is short for NGPU have seen it. NGL is short for NGPU layers or the number of GPU layers. How layers or the number of GPU layers. How layers or the number of GPU layers. How many of those layers get copied into many of those layers get copied into many of those layers get copied into VRAM? So again, I swept it. At zero VRAM? So again, I swept it. At zero VRAM? So again, I swept it. At zero layers, I'm looking over here because layers, I'm looking over here because layers, I'm looking over here because that's where I have my chart, by the that's where I have my chart, by the that's where I have my chart, by the way. At zero layers, it's 4.54 tokens a way. At zero layers, it's 4.54 tokens a way. At zero layers, it's 4.54 tokens a second. At 48 layers, we're up to almost second. At 48 layers, we're up to almost second. At 48 layers, we're up to almost 11. 56 layers, almost 14. It climbs 11. 56 layers, almost 14. It climbs 11. 56 layers, almost 14. It climbs pretty smoothly, the way you'd expect, pretty smoothly, the way you'd expect, pretty smoothly, the way you'd expect, right? Until right? Until right? Until 60 layers. Uh-oh. 0.82 60 layers. Uh-oh. 0.82 60 layers. Uh-oh. 0.82 tokens per second here. Yeah, that's one tokens per second here. Yeah, that's one tokens per second here. Yeah, that's one token every 1 and 1/2 seconds. You can token every 1 and 1/2 seconds. You can token every 1 and 1/2 seconds. You can go get up, make coffee, come back, have go get up, make coffee, come back, have go get up, make coffee, come back, have your cake, and it'll still be typing.

your cake, and it'll still be typing. your cake, and it'll still be typing. Yeah. 56 to 60, four more layers, and Yeah. 56 to 60, four more layers, and Yeah. 56 to 60, four more layers, and it's 17 times slower. There's no warning it's 17 times slower. There's no warning it's 17 times slower. There's no warning and there's no slope. I mean, it is kind and there's no slope. I mean, it is kind and there's no slope. I mean, it is kind of a it's a pretty steep slope. It's a of a it's a pretty steep slope. It's a of a it's a pretty steep slope. It's a wall because the driver is now dragging wall because the driver is now dragging wall because the driver is now dragging those few layers in and out every single those few layers in and out every single those few layers in and out every single word it creates. This was before I word it creates. This was before I word it creates. This was before I realized that I plugged too big a model realized that I plugged too big a model realized that I plugged too big a model in. So, I ran it three times before I in. So, I ran it three times before I in. So, I ran it three times before I realized what happened. Now, before I realized what happened. Now, before I realized what happened. Now, before I would always use NGL99 to do as much would always use NGL99 to do as much would always use NGL99 to do as much offload as possible, but sometimes you offload as possible, but sometimes you offload as possible, but sometimes you don't want to do that. In this case, don't want to do that. In this case, don't want to do that. In this case, that's one of those cases. Look at the that's one of those cases. Look at the that's one of those cases. Look at the top of the ladder here, ngl99. And top of the ladder here, ngl99. And top of the ladder here, ngl99. And that's the flag you'd use with Llama CPP that's the flag you'd use with Llama CPP that's the flag you'd use with Llama CPP in most tutorials that you'll see out in most tutorials that you'll see out in most tutorials that you'll see out there when you copy and paste the there when you copy and paste the there when you copy and paste the commands. Essentially, that means use commands. Essentially, that means use commands. Essentially, that means use all the layers. And what it does is it all the layers. And what it does is it all the layers. And what it does is it overrides Llama CPP's built-in overrides Llama CPP's built-in overrides Llama CPP's built-in autofitting. If you leave the flag out autofitting. If you leave the flag out autofitting. If you leave the flag out and let Llama CPP just automatically do and let Llama CPP just automatically do and let Llama CPP just automatically do it, we're up to 10.48 it, we're up to 10.48 it, we're up to 10.48 tokens per second. But if we handtune it tokens per second. But if we handtune it tokens per second. But if we handtune it at 56, we're up to 14, and that's pretty at 56, we're up to 14, and that's pretty at 56, we're up to 14, and that's pretty big. So that one flag can actually cost big. So that one flag can actually cost big. So that one flag can actually cost you 16 times on a model that doesn't you 16 times on a model that doesn't you 16 times on a model that doesn't fit. Now, auto is not terrible, but just fit. Now, auto is not terrible, but just fit. Now, auto is not terrible, but just know that you could get better know that you could get better know that you could get better performance if you tune. However, both performance if you tune. However, both performance if you tune. However, both of those are still workarounds for a of those are still workarounds for a of those are still workarounds for a model that's 1.7 gigs too big. So, what model that's 1.7 gigs too big. So, what model that's 1.7 gigs too big. So, what if I made the model smaller?

So, I drop one quantization level, Q4KM, So, I drop one quantization level, Q4KM, which is 18 gigs, down to IQ4 XS, 13 which is 18 gigs, down to IQ4 XS, 13 which is 18 gigs, down to IQ4 XS, 13 gigs, and it's still four bits per gigs, and it's still four bits per gigs, and it's still four bits per number. It's just a newer recipe that number. It's just a newer recipe that number. It's just a newer recipe that packs them a little bit tighter. And now packs them a little bit tighter. And now packs them a little bit tighter. And now the whole thing lives inside 16 gigs of the whole thing lives inside 16 gigs of the whole thing lives inside 16 gigs of VRAM. This dock is actually pretty cool. VRAM. This dock is actually pretty cool. VRAM. This dock is actually pretty cool. Not only does it have a graphics card in Not only does it have a graphics card in Not only does it have a graphics card in there, but it gives you SD card, USBC, there, but it gives you SD card, USBC, there, but it gives you SD card, USBC, Ethernet, display port, two HDMIs, two Ethernet, display port, two HDMIs, two Ethernet, display port, two HDMIs, two USBAs, and another USBC on the back. USBAs, and another USBC on the back. USBAs, and another USBC on the back. It's pretty cool. Anyway, back to It's pretty cool. Anyway, back to It's pretty cool. Anyway, back to business. Prompt processing or reading business. Prompt processing or reading business. Prompt processing or reading if you want to call it that. 474 12 up if you want to call it that. 474 12 up if you want to call it that. 474 12 up to 943 now. Token generation or TG up to to 943 now. Token generation or TG up to to 943 now. Token generation or TG up to 27 from 14. Both roughly double. So that 27 from 14. Both roughly double. So that 27 from 14. Both roughly double. So that one little change, just downloading a one little change, just downloading a one little change, just downloading a new quantization, same model, was worth new quantization, same model, was worth new quantization, same model, was worth more than the entire layer splitting more than the entire layer splitting more than the entire layer splitting sweep. Now, I know what you're going to sweep. Now, I know what you're going to sweep. Now, I know what you're going to say in the comments. You're going to say in the comments. You're going to say in the comments. You're going to say, "Alex, you just made it dumber." say, "Alex, you just made it dumber." say, "Alex, you just made it dumber." And that's a fair point, especially And that's a fair point, especially And that's a fair point, especially because in my videos, I've been talking because in my videos, I've been talking because in my videos, I've been talking about how the more you squeeze, the more about how the more you squeeze, the more about how the more you squeeze, the more you quantize, you quantize, you quantize, the less precision you get and therefore the less precision you get and therefore the less precision you get and therefore you get worse quality. Right? I even you get worse quality. Right? I even you get worse quality. Right? I even made a whole video of a sweep going from made a whole video of a sweep going from made a whole video of a sweep going from full precision all the way down to Q2 full precision all the way down to Q2 full precision all the way down to Q2 and Q1. And saying things like, "Oh, it and Q1. And saying things like, "Oh, it and Q1. And saying things like, "Oh, it feels fine." is not really an answer.

feels fine." is not really an answer. feels fine." is not really an answer. So, I needed to test it. The tool that So, I needed to test it. The tool that So, I needed to test it. The tool that we use for that is perplexity, a we use for that is perplexity, a we use for that is perplexity, a surprise score, if you will. You surprise score, if you will. You surprise score, if you will. You basically hand a model text that it's basically hand a model text that it's basically hand a model text that it's never seen and ask at every single word never seen and ask at every single word never seen and ask at every single word how shocked it is or how surprised it is how shocked it is or how surprised it is how shocked it is or how surprised it is by that word. Smaller is better. And by that word. Smaller is better. And by that word. Smaller is better. And after measuring perplexity using Wikiex after measuring perplexity using Wikiex after measuring perplexity using Wikiex 2 plain English Pros, both models come 2 plain English Pros, both models come 2 plain English Pros, both models come at about 6.8. In fact, the smaller one at about 6.8. In fact, the smaller one at about 6.8. In fact, the smaller one is actually a hair better. I mean, it's is actually a hair better. I mean, it's is actually a hair better. I mean, it's just noise basically. So, it's just a just noise basically. So, it's just a just noise basically. So, it's just a rounding error, but still, it looks rounding error, but still, it looks rounding error, but still, it looks better on paper. So, that's 4.5 gigs better on paper. So, that's 4.5 gigs better on paper. So, that's 4.5 gigs smaller and nearly twice as fast with no smaller and nearly twice as fast with no smaller and nearly twice as fast with no quality cost that I can measure. Fitting quality cost that I can measure. Fitting quality cost that I can measure. Fitting beats speed. But let's be clear about beats speed. But let's be clear about beats speed. But let's be clear about what this test covers. These two files what this test covers. These two files what this test covers. These two files come from two different quantizers. So come from two different quantizers. So come from two different quantizers. So what I actually tested was these two what I actually tested was these two what I actually tested was these two files, not IQ4 versus Q4 in general. And files, not IQ4 versus Q4 in general. And files, not IQ4 versus Q4 in general. And in this case, perplexity that I measured in this case, perplexity that I measured in this case, perplexity that I measured is just next token prediction on English is just next token prediction on English is just next token prediction on English pros. It won't detect damage to code or pros. It won't detect damage to code or pros. It won't detect damage to code or arithmetic or long reasoning chains. So arithmetic or long reasoning chains. So arithmetic or long reasoning chains. So that puts a limit on the risk, but it that puts a limit on the risk, but it that puts a limit on the risk, but it doesn't get rid of it. Test your own doesn't get rid of it. Test your own doesn't get rid of it. Test your own workload yourself. As I always say, workload yourself. As I always say, workload yourself. As I always say, right tool for the right job, right?

right tool for the right job, right? right tool for the right job, right? Even different code bases, different Even different code bases, different Even different code bases, different languages act differently with different languages act differently with different languages act differently with different models. Now the model fits, the GPU's models. Now the model fits, the GPU's models. Now the model fits, the GPU's got everything it needs and is fast. got everything it needs and is fast. got everything it needs and is fast. There is one thing left that I hadn't There is one thing left that I hadn't There is one thing left that I hadn't questioned and that's this dock and that questioned and that's this dock and that questioned and that's this dock and that proprietary connection with that high proprietary connection with that high proprietary connection with that high quality high-speed quality high-speed quality high-speed PCI. Is it PCIe? It is PCIe, but it's PCI. Is it PCIe? It is PCIe, but it's PCI. Is it PCIe? It is PCIe, but it's not PCIe 5. It's PCIe 4 when you not PCIe 5. It's PCIe 4 when you not PCIe 5. It's PCIe 4 when you actually hook it up. Still pretty good. actually hook it up. Still pretty good. actually hook it up. Still pretty good. The Kadas Mind Link is what they call The Kadas Mind Link is what they call The Kadas Mind Link is what they call it. It's PCIe 4.0 by 8, roughly 15.8 it. It's PCIe 4.0 by 8, roughly 15.8 it. It's PCIe 4.0 by 8, roughly 15.8 gigabytes a second. This is the headline gigabytes a second. This is the headline gigabytes a second. This is the headline feature, and it's probably the reason feature, and it's probably the reason feature, and it's probably the reason you'd buy it over a generic box or you'd buy it over a generic box or you'd buy it over a generic box or something like this where you have a something like this where you have a something like this where you have a dock. This one's got Oculink and USB 4. dock. This one's got Oculink and USB 4. dock. This one's got Oculink and USB 4. And you buy a separate GPU. This is a And you buy a separate GPU. This is a And you buy a separate GPU. This is a 5060 Ti. And then you hook them up 5060 Ti. And then you hook them up 5060 Ti. And then you hook them up together. Not only is it cleaner, together. Not only is it cleaner, together. Not only is it cleaner, but also is it faster? Well, I kind of but also is it faster? Well, I kind of but also is it faster? Well, I kind of wanted to answer that. How much of that wanted to answer that. How much of that wanted to answer that. How much of that connection pipe does generating text connection pipe does generating text connection pipe does generating text actually need? Loading the model moves actually need? Loading the model moves actually need? Loading the model moves real data. 17 gigs in about 3 seconds.

real data. 17 gigs in about 3 seconds. real data. 17 gigs in about 3 seconds. But once the weights already live in But once the weights already live in But once the weights already live in there, in this chart, I called it there, in this chart, I called it there, in this chart, I called it resident. Generating text peaks at 19 resident. Generating text peaks at 19 resident. Generating text peaks at 19 megabytes a second and averages 2 megabytes a second and averages 2 megabytes a second and averages 2 megabytes a second. megabytes. That's megabytes a second. megabytes. That's megabytes a second. megabytes. That's about a hundth of a percent of what that about a hundth of a percent of what that about a hundth of a percent of what that link is capable of. You'd have to make link is capable of. You'd have to make link is capable of. You'd have to make this connection about 8,000 times slower this connection about 8,000 times slower this connection about 8,000 times slower before it became the bottleneck. Even before it became the bottleneck. Even before it became the bottleneck. Even Thunderbolt on paper is about 5 times Thunderbolt on paper is about 5 times Thunderbolt on paper is about 5 times slower than this. It's slower, but it's slower than this. It's slower, but it's slower than this. It's slower, but it's five, not 8,000. I spent a lot of time five, not 8,000. I spent a lot of time five, not 8,000. I spent a lot of time obsessing over this 15.8 GB pipe, and obsessing over this 15.8 GB pipe, and obsessing over this 15.8 GB pipe, and the thing I was measuring just needed 2 the thing I was measuring just needed 2 the thing I was measuring just needed 2 megabytes. Now, I've been throwing megabytes. Now, I've been throwing megabytes. Now, I've been throwing numbers at you and telling you to trust numbers at you and telling you to trust numbers at you and telling you to trust them. So, I owe you a little more them. So, I owe you a little more them. So, I owe you a little more explanation here. There have been a explanation here. There have been a explanation here. There have been a couple of times where I found the couple of times where I found the couple of times where I found the numbers to be wrong. This is just part numbers to be wrong. This is just part numbers to be wrong. This is just part of uh a YouTuber testing things. Uh of uh a YouTuber testing things. Uh of uh a YouTuber testing things. Uh yeah, I'm trying to get better, okay? yeah, I'm trying to get better, okay? yeah, I'm trying to get better, okay? But I wanted to reveal this to you But I wanted to reveal this to you But I wanted to reveal this to you because you might get stuck in the same because you might get stuck in the same because you might get stuck in the same thing and I don't want you to. thing and I don't want you to. thing and I don't want you to. Initially, when I was measuring this, Initially, when I was measuring this, Initially, when I was measuring this, there was an invisible process that there was an invisible process that there was an invisible process that basically haved the result I took. I basically haved the result I took. I basically haved the result I took. I would kill a benchmark and Llama Bench would kill a benchmark and Llama Bench would kill a benchmark and Llama Bench stayed resident, holding on to about stayed resident, holding on to about stayed resident, holding on to about 18.5 GB. More than a quarter of the 18.5 GB. More than a quarter of the 18.5 GB. More than a quarter of the machine's available memory just gone machine's available memory just gone machine's available memory just gone invisibly. Of course, every reading invisibly. Of course, every reading invisibly. Of course, every reading after that came back less than half of after that came back less than half of after that came back less than half of what it should have been, and nothing in what it should have been, and nothing in what it should have been, and nothing in the output said why. So, I basically the output said why. So, I basically the output said why. So, I basically threw away a whole day of testing. So, threw away a whole day of testing. So, threw away a whole day of testing. So, if you're going to be doing this on your if you're going to be doing this on your if you're going to be doing this on your own code, make sure you reboot or just own code, make sure you reboot or just own code, make sure you reboot or just kill the process intentionally. And also kill the process intentionally. And also kill the process intentionally. And also the moment a second GPU showed up, Open

the moment a second GPU showed up, Open the moment a second GPU showed up, Open Veno silently ran on the CPU. That Veno silently ran on the CPU. That Veno silently ran on the CPU. That caused me a 30 times error because for caused me a 30 times error because for caused me a 30 times error because for some reason the environment variable some reason the environment variable some reason the environment variable wants GPU.0 wants GPU.0 wants GPU.0 with0 in it. with0 in it. with0 in it. Come on. But I got that all figured out. Come on. But I got that all figured out. Come on. But I got that all figured out. But finally the machine died. Um But finally the machine died. Um But finally the machine died. Um and it took me a while to realize why. and it took me a while to realize why. and it took me a while to realize why. See, this box is uh it also has a UPS, a See, this box is uh it also has a UPS, a See, this box is uh it also has a UPS, a battery that lives inside this box. When battery that lives inside this box. When battery that lives inside this box. When you're running it and you're connected you're running it and you're connected you're running it and you're connected to power, you don't know that it's to power, you don't know that it's to power, you don't know that it's running on the battery or how much running on the battery or how much running on the battery or how much battery it's using or how much power battery it's using or how much power battery it's using or how much power it's using. It's a good thing because if it's using. It's a good thing because if it's using. It's a good thing because if you accidentally disconnect the power you accidentally disconnect the power you accidentally disconnect the power cable is not going to lose everything. cable is not going to lose everything. cable is not going to lose everything. It's just going to keep running and it It's just going to keep running and it It's just going to keep running and it gives you a few minutes of run time, gives you a few minutes of run time, gives you a few minutes of run time, which is really nice. However, if you which is really nice. However, if you which is really nice. However, if you plug in a power cord that's just lower plug in a power cord that's just lower plug in a power cord that's just lower than the utilization, I was really than the utilization, I was really than the utilization, I was really pushing this box and I had a 30 watt pushing this box and I had a 30 watt pushing this box and I had a 30 watt charging cable plugged into it, which charging cable plugged into it, which charging cable plugged into it, which slowly drained the battery until the slowly drained the battery until the slowly drained the battery until the machine just died. And I was like, machine just died. And I was like, machine just died. And I was like, "What? What happened?" So, once I "What? What happened?" So, once I "What? What happened?" So, once I plugged a 100 watt cable into it, it plugged a 100 watt cable into it, it plugged a 100 watt cable into it, it came back on and everything was fine.

came back on and everything was fine. came back on and everything was fine. It'd be nice to have two little lights It'd be nice to have two little lights It'd be nice to have two little lights on the front that say whether you're on the front that say whether you're on the front that say whether you're using the battery or not or the power or using the battery or not or the power or using the battery or not or the power or not, just so that people are aware. So not, just so that people are aware. So not, just so that people are aware. So then the uh Austar dock that I tried, I then the uh Austar dock that I tried, I then the uh Austar dock that I tried, I kind of wanted to get the numbers for kind of wanted to get the numbers for kind of wanted to get the numbers for this as well, but it crashed twice both this as well, but it crashed twice both this as well, but it crashed twice both times moving model into VRAMm of the times moving model into VRAMm of the times moving model into VRAMm of the card. The display driver stopped card. The display driver stopped card. The display driver stopped responding and Windows just gave up. I responding and Windows just gave up. I responding and Windows just gave up. I should and I know you will mention this should and I know you will mention this should and I know you will mention this in the comments. I should also test in the comments. I should also test in the comments. I should also test Linux on this thing, but I was a little Linux on this thing, but I was a little Linux on this thing, but I was a little worried about the special drivers to worried about the special drivers to worried about the special drivers to support that link. So I didn't do that support that link. So I didn't do that support that link. So I didn't do that yet, but I may do that. Let me know in yet, but I may do that. Let me know in yet, but I may do that. Let me know in the comments if you're interested in the comments if you're interested in the comments if you're interested in seeing that. So, even though I wanted to seeing that. So, even though I wanted to seeing that. So, even though I wanted to do a comparison between the proprietary do a comparison between the proprietary do a comparison between the proprietary dock link versus the speed of a dock link versus the speed of a dock link versus the speed of a Thunderbolt dock to demonstrate that it Thunderbolt dock to demonstrate that it Thunderbolt dock to demonstrate that it probably doesn't make much of a probably doesn't make much of a probably doesn't make much of a difference, if you're gaming on it, that difference, if you're gaming on it, that difference, if you're gaming on it, that might make a difference, but for LLMs, might make a difference, but for LLMs, might make a difference, but for LLMs, probably not. All right, here's every probably not. All right, here's every probably not. All right, here's every configuration I tried on one chart from configuration I tried on one chart from configuration I tried on one chart from slowest to fastest. Couple things that slowest to fastest. Couple things that slowest to fastest. Couple things that really stood out. CPU only under five really stood out. CPU only under five really stood out. CPU only under five tokens a second. Integrated GPU 5.5. If tokens a second. Integrated GPU 5.5. If tokens a second. Integrated GPU 5.5. If you add the draft model, you get up to you add the draft model, you get up to you add the draft model, you get up to 11 with a discrete GPU split 14 tokens 11 with a discrete GPU split 14 tokens 11 with a discrete GPU split 14 tokens per second. And then fully resident per second. And then fully resident per second. And then fully resident inside this thing, we get 27. And if you inside this thing, we get 27. And if you inside this thing, we get 27. And if you add a draft model to that, we get up to add a draft model to that, we get up to add a draft model to that, we get up to 44. Huge range, nine times faster on the 44. Huge range, nine times faster on the 44. Huge range, nine times faster on the same box with the same model and same box with the same model and same box with the same model and prefill, which is not on this chart, it prefill, which is not on this chart, it prefill, which is not on this chart, it goes even further. 17 tokens per second goes even further. 17 tokens per second goes even further. 17 tokens per second to over 900 56 times. Remember how at

to over 900 56 times. Remember how at to over 900 56 times. Remember how at the start I said running it wasn't the the start I said running it wasn't the the start I said running it wasn't the surprising part? This is so making the surprising part? This is so making the surprising part? This is so making the model fit, that's free. Turning on the model fit, that's free. Turning on the model fit, that's free. Turning on the draft model, that's also free. Deleting draft model, that's also free. Deleting draft model, that's also free. Deleting a flag from the tutorial and tuning it a flag from the tutorial and tuning it a flag from the tutorial and tuning it yourself, that's free. But buying a GPU, yourself, that's free. But buying a GPU, yourself, that's free. But buying a GPU, that one is not free. This dock doesn't that one is not free. This dock doesn't that one is not free. This dock doesn't come with the machine. You'd have to buy come with the machine. You'd have to buy come with the machine. You'd have to buy it separately. I believe it's $1,000, it separately. I believe it's $1,000, it separately. I believe it's $1,000, but I'll link to the deals down below if but I'll link to the deals down below if but I'll link to the deals down below if you're curious. But because of my you're curious. But because of my you're curious. But because of my failures with an external dock, I would failures with an external dock, I would failures with an external dock, I would actually recommend you get the actual actually recommend you get the actual actually recommend you get the actual Kadas dock if you're buying this Kadas dock if you're buying this Kadas dock if you're buying this machine. So if you're doing this machine. So if you're doing this machine. So if you're doing this yourself, check whether your model fits yourself, check whether your model fits yourself, check whether your model fits before you buy anything. The biggest win before you buy anything. The biggest win before you buy anything. The biggest win on the list and it cost you just a on the list and it cost you just a on the list and it cost you just a download. Also, don't just paste NGL99. download. Also, don't just paste NGL99. download. Also, don't just paste NGL99. Oh, and turn on the draft model only if Oh, and turn on the draft model only if Oh, and turn on the draft model only if you're the only person using the box you're the only person using the box you're the only person using the box because if it's more than two people at because if it's more than two people at because if it's more than two people at once, turn it off. because past that its once, turn it off. because past that its once, turn it off. because past that its guesses are competing with real users guesses are competing with real users guesses are competing with real users for the same pass and real users will for the same pass and real users will for the same pass and real users will win. But here's a short version. Buy win. But here's a short version. Buy win. But here's a short version. Buy VRAMm not bandwidth. Now I did have VRAMm not bandwidth. Now I did have VRAMm not bandwidth. Now I did have successful experiments with an external successful experiments with an external successful experiments with an external GPU. If you want to see what happens GPU. If you want to see what happens GPU. If you want to see what happens with that, watch this video next. Thanks with that, watch this video next. Thanks with that, watch this video next. Thanks for watching and I'll see you next time.