Building AI apps on Local AI & Kubernetes

You have to bear with me for a little You have to bear with me for a little bit as I'm setting this all up. bit as I'm setting this all up. bit as I'm setting this all up. But let's see. I do have a chat now. But let's see. I do have a chat now. But let's see. I do have a chat now. Chat settings. Chat settings. Chat settings. I have just purchased I'm sort of figuring this out. Oh, some I'm sort of figuring this out. Oh, some people are joining in already. Hello. Let's see. Let's see. Embed or launch chat. Embed or launch chat. Embed or launch chat. Here we go. Move Move to new window. to new window. to new window. So, this will be my chat window. I've been using Zen browser and loving I've been using Zen browser and loving it. Ever tried it? No, but this is it. Ever tried it? No, but this is it. Ever tried it? No, but this is definitely the browser that is on my definitely the browser that is on my definitely the browser that is on my list, actually. list, actually. list, actually. So, I'm glad that you're saying that you So, I'm glad that you're saying that you So, I'm glad that you're saying that you like it.

Why is it saying that my bitrate is Why is it saying that my bitrate is 8,000? 8,000? 8,000? I'm not even sure if I can change that. No, I'm literally streaming at 4K. So, No, I'm literally streaming at 4K. So, how can it say that how can it say that how can it say that my bitrate is 6,000? Yes, I am on the walking pad. It's a new Yes, I am on the walking pad. It's a new addition to my to my setup here. Actually, I'm seeing the chat on top. I Actually, I'm seeing the chat on top. I don't like that. I want to have the chat don't like that. I want to have the chat don't like that. I want to have the chat aligned from the bottom. aligned from the bottom. aligned from the bottom. Let me see. Ooh, I have some some settings too. Ooh, I have some some settings too. Wow.

Oh, that's pretty cool. Oh, that's pretty cool. But maybe a bit hard to read, though. Fortnite. Fortnite. Overwatch. Um okay. Um okay. So, How did I do that again? How did I do that again? Chat. Chat. Chat. >> [snorts] >> [snorts] >> [snorts] >> Chat settings.

Yeah, so now it's aligned to the bottom. Yeah, so now it's aligned to the bottom. That's what I wanted. Okay. That's what I wanted. Okay. That's what I wanted. Okay. So, this one can go here. looks like we are fully live now. 17 looks like we are fully live now. 17 people tuning in. Are we live on X too? Let's check that Are we live on X too? Let's check that first. first. first. x.com. Yes, I am producing Fedora's way. That's Yes, I am producing Fedora's way. That's right. Okay. Okay. And actually, I think the place where And actually, I think the place where And actually, I think the place where we're going to be spending most of our we're going to be spending most of our we're going to be spending most of our time is going to be the terminal, time is going to be the terminal, time is going to be the terminal, actually. So, I'll put you guys here beside my I'll put you guys here beside my terminal.

Hello from Berlin. Hello from Berlin. Welcome from Berlin. Welcome from Berlin. Welcome from Berlin. Very good to see. Some buzz mount sound coming in the mic Some buzz mount sound coming in the mic and the volume is low. and the volume is low. and the volume is low. Thank you for mentioning that. Let me Thank you for mentioning that. Let me Thank you for mentioning that. Let me see. Okay, so as you just saw, my OBS just Okay, so as you just saw, my OBS just crashed. But there's buzz sounds coming in the But there's buzz sounds coming in the mic. Let me Let me mic. Let me Let me mic. Let me Let me try to fix that.

All right. All right. Is this any better? Can you please let me know Can you please let me know if this sounds better? I may also have to increase the gain a I may also have to increase the gain a bit if it's too soft. Am on Gupta. Can you let me know if this Am on Gupta. Can you let me know if this is better? Looks and sounds okay to me. Okay. Looks and sounds okay to me. Okay. It is good now. Okay. Well, thank you It is good now. Okay. Well, thank you It is good now. Okay. Well, thank you for pointing that out. Like for pointing that out. Like for pointing that out. Like I am um I am um I am um Now that we have some people Now that we have some people Now that we have some people tuning in, I can talk a little bit about tuning in, I can talk a little bit about tuning in, I can talk a little bit about what my plan here is. what my plan here is. what my plan here is. So, I have been So, I have been So, I have been working on an application working on an application working on an application for my students, for my students, for my students, which I call CubeCraft AI.

which I call CubeCraft AI. which I call CubeCraft AI. And this is a custom AI application that And this is a custom AI application that And this is a custom AI application that I'm that I built myself. I'm that I built myself. I'm that I built myself. And when I'm running this, And when I'm running this, And when I'm running this, this actually this actually this actually uh when I ask, for example, should I use uh when I ask, for example, should I use uh when I ask, for example, should I use Proxmox for my home lab? Proxmox for my home lab? Proxmox for my home lab? It is going to be querying It is going to be querying It is going to be querying two over two years of coaching calls two over two years of coaching calls two over two years of coaching calls that I have done with all of my paid that I have done with all of my paid that I have done with all of my paid students. students. students. It's querying all of my YouTube videos It's querying all of my YouTube videos It's querying all of my YouTube videos that I've created. that I've created. that I've created. And then it's going to give an answer And then it's going to give an answer And then it's going to give an answer based on those based on those based on those um um um the answers that I've given in my the answers that I've given in my the answers that I've given in my content. content. content. So, I have these, you know, specific So, I have these, you know, specific So, I have these, you know, specific takes on things and I have takes on things and I have takes on things and I have my own special approach in teaching my own special approach in teaching my own special approach in teaching people. people. people. And And And the AI so far is doing a really good job the AI so far is doing a really good job the AI so far is doing a really good job at capturing that. at capturing that. at capturing that. So, if I So, if I So, if I create a new chat and ask, what is the create a new chat and ask, what is the create a new chat and ask, what is the best distro best distro best distro uh uh uh the best Linux distro the best Linux distro the best Linux distro uh for my for a DevOps career?

uh for my for a DevOps career? uh for my for a DevOps career? For example, then I expect that it will For example, then I expect that it will For example, then I expect that it will say Arch Linux. say Arch Linux. say Arch Linux. Uh yeah, here. So, For building Uh yeah, here. So, For building Uh yeah, here. So, For building foundational skills, he mentions that foundational skills, he mentions that foundational skills, he mentions that his own Linux admin skills were 100% his own Linux admin skills were 100% his own Linux admin skills were 100% learned using Arch Linux. learned using Arch Linux. learned using Arch Linux. Um Um Um Can be a good idea. Can be a good idea. Can be a good idea. Little bit bloated Ubuntu desktop. Okay, Little bit bloated Ubuntu desktop. Okay, Little bit bloated Ubuntu desktop. Okay, that's not completely true. But But yeah, he does picks up he did pick up yeah, he does picks up he did pick up yeah, he does picks up he did pick up that I switched to Fedora, for example, that I switched to Fedora, for example, that I switched to Fedora, for example, but this is correct, though. but this is correct, though. but this is correct, though. So, and it then references my the some So, and it then references my the some So, and it then references my the some some videos that I've done with links. some videos that I've done with links. some videos that I've done with links. So, this is what then what my paid So, this is what then what my paid So, this is what then what my paid students can use to get help 24/7. And uh no, this is only for my my And uh no, this is only for my my private students. So, my my paid private students. So, my my paid private students. So, my my paid students, they are the ones who are students, they are the ones who are students, they are the ones who are getting access to this application. getting access to this application. getting access to this application. And this is what I'm building. And this is what I'm building. And this is what I'm building. But the cool thing is is that this is But the cool thing is is that this is But the cool thing is is that this is running on a kubernetes cluster on my running on a kubernetes cluster on my running on a kubernetes cluster on my own local kubernetes cluster.

own local kubernetes cluster. own local kubernetes cluster. So I have a cluster running So I have a cluster running So I have a cluster running Talos Linux Talos Linux Talos Linux with a GPU. with a GPU. with a GPU. And I And I And I am running this as am running this as am running this as a two node kubernetes cluster a two node kubernetes cluster a two node kubernetes cluster where my So when I send this request where my So when I send this request where my So when I send this request it is actually going to be it is actually going to be it is actually going to be sent to sent to sent to the the API server on the kubernetes the the API server on the kubernetes the the API server on the kubernetes cluster which then queries a local AI cluster which then queries a local AI cluster which then queries a local AI model running in the cluster model running in the cluster model running in the cluster using the GPU. I'm using a 4090 GPU. using the GPU. I'm using a 4090 GPU. using the GPU. I'm using a 4090 GPU. And And And this is where it's running. So it's this is where it's running. So it's this is where it's running. So it's running completely local. running completely local. running completely local. And I'm also building this myself. So of And I'm also building this myself. So of And I'm also building this myself. So of course using cloud code and AI. course using cloud code and AI. course using cloud code and AI. So So So as I now have a working setup going, I as I now have a working setup going, I as I now have a working setup going, I know that I want to keep going with know that I want to keep going with know that I want to keep going with this. this. this. And I am planning to document this And I am planning to document this And I am planning to document this process basically on these live streams process basically on these live streams process basically on these live streams etc.

etc. etc. So I am So I am So I am planning to keep I'm basically going to planning to keep I'm basically going to planning to keep I'm basically going to live stream much more live stream much more live stream much more more than I have been doing more than I have been doing more than I have been doing so far. so far. so far. And I want Yeah, I basically have the And I want Yeah, I basically have the And I want Yeah, I basically have the goal of doing more live streams and goal of doing more live streams and goal of doing more live streams and documenting this process. documenting this process. documenting this process. Because I also want to I'm learning so Because I also want to I'm learning so Because I also want to I'm learning so much here that much here that much here that yeah, my idea is to start sharing that yeah, my idea is to start sharing that yeah, my idea is to start sharing that too. too. too. And also I have a nice trip planned And also I have a nice trip planned And also I have a nice trip planned in two weeks from now. in two weeks from now. in two weeks from now. And then the idea is that I'm also going And then the idea is that I'm also going And then the idea is that I'm also going to try to do some to try to do some to try to do some live streaming from the road. So you live streaming from the road. So you live streaming from the road. So you guys can also see how what it's like to guys can also see how what it's like to guys can also see how what it's like to live like a sovereign craftsman like me. live like a sovereign craftsman like me. live like a sovereign craftsman like me. And being able to And being able to And being able to work from the road and to do some live work from the road and to do some live work from the road and to do some live streaming and yeah, basically get a a streaming and yeah, basically get a a streaming and yeah, basically get a a sense of how I live my life. sense of how I live my life. sense of how I live my life. So that is what I am planning to do.

So that is what I am planning to do. So that is what I am planning to do. And yeah, with that being said, I have a And yeah, with that being said, I have a And yeah, with that being said, I have a big feature that big feature that big feature that um um um I have scoped out and I'm basically I have scoped out and I'm basically I have scoped out and I'm basically going to just going to just going to just um um um keep working on this while I'm on keep working on this while I'm on keep working on this while I'm on stream. stream. stream. So So So the what I'm the feature I'm currently the what I'm the feature I'm currently the what I'm the feature I'm currently working on is I it is currently quite working on is I it is currently quite working on is I it is currently quite painful for me to switch local models painful for me to switch local models painful for me to switch local models on the cluster because it's very hard on the cluster because it's very hard on the cluster because it's very hard for to for to for to because I only have one GPU, I can only because I only have one GPU, I can only because I only have one GPU, I can only run one model at a time. run one model at a time. run one model at a time. And as I was trying another local model And as I was trying another local model And as I was trying another local model the other day the other day the other day I was running into quite a fair bit of I was running into quite a fair bit of I was running into quite a fair bit of issues while doing that. issues while doing that. issues while doing that. So I am now building a feature that is So I am now building a feature that is So I am now building a feature that is going to make that going to make that going to make that um um um a lot easier. a lot easier. a lot easier. So let's see. So let's see. So let's see. Can you Can you Can you restate that full message restate that full message restate that full message you sent here?

you sent here? you sent here? The output got garbled. Do you offer localized prices in your Do you offer localized prices in your mentorship program? Unfortunately, I mentorship program? Unfortunately, I mentorship program? Unfortunately, I cannot. cannot. cannot. We do have some payment options We do have some payment options We do have some payment options available. available. available. But otherwise um there is no localized pricing. there is no localized pricing. Okay, so all five are all five PRs were Okay, so all five are all five PRs were Okay, so all five are all five PRs were implemented and double reviewed. implemented and double reviewed. implemented and double reviewed. So So So it has been working on this feature it has been working on this feature it has been working on this feature yesterday and the day before yesterday yesterday and the day before yesterday yesterday and the day before yesterday and yesterday I didn't have time. and yesterday I didn't have time. and yesterday I didn't have time. So now I am continuing where I was So now I am continuing where I was So now I am continuing where I was basically. basically. basically. So let me see. Merge order plus your So let me see. Merge order plus your So let me see. Merge order plus your cluster work between PRs. So first it's cluster work between PRs. So first it's cluster work between PRs. So first it's we're going to set up a we're going to set up a we're going to set up a new download feature that is downloading new download feature that is downloading new download feature that is downloading them to a persistent volume from them to a persistent volume from them to a persistent volume from um um um Hugging Face.

Extract the old secrets, create new Extract the old secrets, create new secrets. secrets. secrets. Okay. Okay. Okay. So I'll have to check out that run book So I'll have to check out that run book So I'll have to check out that run book there. Docs migrations. Docs migrations. Docs Oh, okay. Yeah, this is a different Oh, okay. Yeah, this is a different branch. Okay. Yeah, so we have to run branch. Okay. Yeah, so we have to run branch. Okay. Yeah, so we have to run some migrations. Depends on 134 merged.

Depends on 134 merged. Add model ID. Okay. So it has given me the the Okay. So it has given me the the the order.

So I will first have to create a token So I will first have to create a token secret. secret. secret. And from that And from that And from that I will first have to I will first have to I will first have to Let's log in to the cluster. And then NSL And then NSL and demo. and demo. and demo. Okay, so this is where the actual pods Okay, so this is where the actual pods Okay, so this is where the actual pods that are running the local. Thanks for all the videos helping to Thanks for all the videos helping to learn Devops Kubernetes. Yeah, you're learn Devops Kubernetes. Yeah, you're learn Devops Kubernetes. Yeah, you're welcome man. welcome man. welcome man. I hope they have been useful to you. Um okay, so I'll need to set up my Um okay, so I'll need to set up my Hugging Face token. So I'm going to Hugging Face token. So I'm going to Hugging Face token. So I'm going to switch to Let's see. Let me get that token up and Let's see. Let me get that token up and running.

I should actually also I should actually also uh Hi from Norway.

Hi from Norway. Hi hi. How are you doing? How are you Hi hi. How are you doing? How are you Hi hi. How are you doing? How are you doing? doing? doing? German Gustav Thorsen. German Gustav Thorsen. German Gustav Thorsen. That is a That is a That is a typical Norwegian name. Yeah. I just spoke some Norwegian to one of I just spoke some Norwegian to one of our channel friends. And I'm In the meantime, I'm just And I'm In the meantime, I'm just figuring out my Hugging Face token here. So bear with me Hugging Face token here. So bear with me here. He speaks Norwegian well. He speaks Norwegian well. Yes. Yes. Yes. I speak very well or

>> All right, so 93 and 1/2% so >> All right, so 93 and 1/2% so According to Yermo, my Norwegian my According to Yermo, my Norwegian my According to Yermo, my Norwegian my Norwegian level is 93 and 1/2% I'll take Norwegian level is 93 and 1/2% I'll take Norwegian level is 93 and 1/2% I'll take it. All right, so I don't have a token yet. All right, so I don't have a token yet. So I will just have to So I will just have to So I will just have to create a new token. I'm sorry. I can't create a new token. I'm sorry. I can't create a new token. I'm sorry. I can't share that obviously, but when I have share that obviously, but when I have share that obviously, but when I have the token I'll be the token I'll be the token I'll be um So I'm logged into hugging face now So I'm logged into hugging face now and and and I want to see my tokens. It's asking me I want to see my tokens. It's asking me I want to see my tokens. It's asking me for a password, but I never set a

for a password, but I never set a for a password, but I never set a password. password. password. So Like why? Like why? I've never set a password here. I've never set a password here. I've never set a password here. So I'll first have to set a password So I'll first have to set a password So I'll first have to set a password then maybe. Okay, well let me just show you what's Okay, well let me just show you what's going on here. going on here. going on here. Um Um Um I'm I'm I'm logging in to hugging face. I will just use the forgot my password. I will just use the forgot my password. And I will then Access tokens.

Access tokens. Create new token. Create new token. Create new token. All right. All right. All right. Um So KCAI So KCAI read read read Create token. Okay, so I finally have my hugging face Okay, so I finally have my hugging face token. token. token. Uh export uh uh hugging face All right, so now I have a hugging face All right, so now I have a hugging face token and I can continue.

Is asking is the stream available Is asking is the stream available afterwards and yes, I'm streaming to afterwards and yes, I'm streaming to afterwards and yes, I'm streaming to to YouTube. What's your experience with K3D? What's your experience with K3D? >> [snorts] >> [snorts] >> [snorts] >> Yes, I love K3D. I've been using it a >> Yes, I love K3D. I've been using it a >> Yes, I love K3D. I've been using it a lot. Is there any hope for coding jobs in Is there any hope for coding jobs in this area because there's so many this area because there's so many this area because there's so many layoffs going on? layoffs going on? layoffs going on? Coding jobs specifically are Yeah, the future does not look bright Yeah, the future does not look bright there. there. there. Like I've been coding this whole app Like I've been coding this whole app Like I've been coding this whole app um With like I'm not a software engineer, With like I'm not a software engineer, right? But I right? But I right? But I um I I have been building this whole entire I have been building this whole entire app just from from scratch and yeah, the app just from from scratch and yeah, the app just from from scratch and yeah, the actual writing code is what these these actual writing code is what these these actual writing code is what these these these large language models are really these large language models are really these large language models are really good at.

good at. good at. So So So like coding jobs, I wouldn't uh I like coding jobs, I wouldn't uh I like coding jobs, I wouldn't uh I wouldn't bet my career on that actually. Coding is basically outsourced at this Coding is basically outsourced at this stage. Okay, so now this is created. Let me Okay, so now this is created. Let me just just just quickly check if that came through quickly check if that came through quickly check if that came through correctly. It's always a little bit tricky when It's always a little bit tricky when you're doing you're doing you're doing um um um working with tokens and credentials working with tokens and credentials working with tokens and credentials while streaming. I have to be very while streaming. I have to be very while streaming. I have to be very careful that I'm careful that I'm careful that I'm I'm not I'm not doing this alone I'm not I'm not doing this alone I'm not I'm not doing this alone anymore. I actually have people anymore. I actually have people anymore. I actually have people watching. watching. watching. So So So the code the code the code Yeah, token looks right. So then I can So then I can I really have to set up a key I really have to set up a key I really have to set up a key um um um to do this properly.

to do this properly. to do this properly. Like a Like a Like a hot key. hot key. hot key. >> [snorts] >> [snorts] >> [snorts] >> Let's create the secret inside of the >> Let's create the secret inside of the >> Let's create the secret inside of the What is it? K get NS? Okay. Okay. What should we learn to survive in this What should we learn to survive in this What should we learn to survive in this world? Well, I recommend you learn this world? Well, I recommend you learn this world? Well, I recommend you learn this the exact skills that you are seeing the exact skills that you are seeing the exact skills that you are seeing right here, right? right here, right? right here, right? AI AI AI Kubernetes, DevOps That is going to be a That is going to be a um um um massive massive massive There's going to be so much work here. There's going to be so much work here. There's going to be so much work here. We can see the credentials through your We can see the credentials through your We can see the credentials through your glasses. Oh my God. glasses. Oh my God. glasses. Oh my God. I think you are right.

That's actually a really good point. That's actually a really good point. >> [laughter] >> [laughter] >> [laughter] >> I have to >> I have to >> I have to Like if you really wanted you could Like if you really wanted you could Like if you really wanted you could probably see them. Fortunately, it was a probably see them. Fortunately, it was a probably see them. Fortunately, it was a read-only key. read-only key. read-only key. But I should actually create a scene But I should actually create a scene But I should actually create a scene here. here. here. Uh add scene Uh add scene Uh add scene which we call nothing. And this should just literally show And this should just literally show nothing. nothing. nothing. Let me see if that works. Yeah. That is actually working. Okay, so Yeah. That is actually working. Okay, so I'll I'll have to create some sort of I'll I'll have to create some sort of I'll I'll have to create some sort of image for that, but now image for that, but now image for that, but now I can at least I can like I can at least I can like I can at least I can like remove my complete screen. Thoughts about N8N? I use it. I'm Thoughts about N8N? I use it. I'm self-hosting it. Works really well. merge merge and then we're going to test to download and then we're going to test to download and then we're going to test to download a model. So, first we're going to merge a model. So, first we're going to merge a model. So, first we're going to merge the PR.

the PR. the PR. Um So, here's the pull request So, here's the pull request uh number 133. uh number 133. uh number 133. So, this one This is basically a script that I use This is basically a script that I use to download the models from Hugging Face to download the models from Hugging Face to download the models from Hugging Face onto a persistent volume. Because I want to pre be able to Because I want to pre be able to pre-pull them instead of my users having pre-pull them instead of my users having pre-pull them instead of my users having to wait for that. to wait for that. to wait for that. So, first we are going to merge this. And And then then then infra download model If I then get switch If I then get switch master master master pull that CD infra CD infra infra download model download model Quen 35B

Okay, so it created like a little script Okay, so it created like a little script here. download the Quen models. download the Quen models. Actually, I should update this. Cuz I was trying to get a Quen model to Cuz I was trying to get a Quen model to run run run uh earlier uh earlier uh earlier with 27B, but that didn't work. with 27B, but that didn't work. with 27B, but that didn't work. Well, actually let's just Right now, I'm Well, actually let's just Right now, I'm Well, actually let's just Right now, I'm just testing to download the models, so just testing to download the models, so just testing to download the models, so I'm just going to I'm just going to I'm just going to uh uh uh Let's just do this one first. Quen 35B. Let's just do this one first. Quen 35B. Let's just do this one first. Quen 35B. We're just testing if the script works We're just testing if the script works We're just testing if the script works at this stage. at this stage. at this stage. So So So um Let's just see how that Let's just see how that performs.

Okay, so this is now downloading if it's Okay, so this is now downloading if it's now creating a job on the now creating a job on the now creating a job on the on the cluster. view the pods in all name spaces view the pods in all name spaces And here we see that there's a job And here we see that there's a job And here we see that there's a job running. running. running. This job is then This job is then This job is then mounting a mounting a mounting a persistent volume claim, which basically persistent volume claim, which basically persistent volume claim, which basically has my has my has my um local models stored on the PVC. local models stored on the PVC. Let me see. Can I see the mounts? Yes.

Let me see. Can I see the mounts? Yes. Let me see. Can I see the mounts? Yes. So, it's now just downloading it. So, it's now just downloading it. So, it's now just downloading it. Let me see if it is Yeah, it seems like it's downloading the Yeah, it seems like it's downloading the tokens. Yeah. tokens. Yeah. tokens. Yeah. Using the token, so that's good. Using the token, so that's good. Using the token, so that's good. All right, so that seems to be working. All right, so that seems to be working. All right, so that seems to be working. Hello Harsh. You are on Twitch. Hello Harsh. You are on Twitch. Hello Harsh. You are on Twitch. Nice to see some people from Twitch Nice to see some people from Twitch Nice to see some people from Twitch tuning in. Uh Uh Yeah, so what I'm building is I I Yeah, so what I'm building is I I Yeah, so what I'm building is I I already explained it earlier in the already explained it earlier in the already explained it earlier in the video, but I am building a custom video, but I am building a custom video, but I am building a custom application for my paid students application for my paid students application for my paid students that has all of my teachings in them, so that has all of my teachings in them, so that has all of my teachings in them, so uh over 2 years of coaching calls, all uh over 2 years of coaching calls, all uh over 2 years of coaching calls, all of my YouTube videos, my courses, etc. of my YouTube videos, my courses, etc. of my YouTube videos, my courses, etc. So, people can ask it questions and it So, people can ask it questions and it So, people can ask it questions and it will answer in my voice and give my will answer in my voice and give my will answer in my voice and give my opinions on on certain topics.

opinions on on certain topics. opinions on on certain topics. And this is all running on a local And this is all running on a local And this is all running on a local Kubernetes cluster with a local GPU all Kubernetes cluster with a local GPU all Kubernetes cluster with a local GPU all being trained and run and served from being trained and run and served from being trained and run and served from Kubernetes. So, that is in short what I Kubernetes. So, that is in short what I Kubernetes. So, that is in short what I am am am what I'm creating. what I'm creating. what I'm creating. So, the model is downloading. It takes So, the model is downloading. It takes So, the model is downloading. It takes some time. some time. some time. But that's good. Do you have any takes on tools to deploy Do you have any takes on tools to deploy resources on Kubernetes? resources on Kubernetes? resources on Kubernetes? Yeah, definitely don't use Terraform. Yeah, definitely don't use Terraform. Yeah, definitely don't use Terraform. Like the industry has decided that Like the industry has decided that Like the industry has decided that GitOps is the way. So Basically Basically anything anything anything that is not GitOps is out of the that is not GitOps is out of the that is not GitOps is out of the question. And you do not use Crossplane for And you do not use Crossplane for GitOps. GitOps. GitOps. Crossplane is to deploy other resources Crossplane is to deploy other resources Crossplane is to deploy other resources from Kubernetes.

Okay, so the down the down It is slowly Okay, so the down the down It is slowly but surely downloading, so that's good. but surely downloading, so that's good. but surely downloading, so that's good. And actually what's interesting is when And actually what's interesting is when And actually what's interesting is when you look at you look at you look at a model I I thought it was like one a model I I thought it was like one a model I I thought it was like one file. Oh, it's gone now. Oh, it's gone now. I can't attach to it, but um I can't attach to it, but um I can't attach to it, but um It's actually a pretty big collection of It's actually a pretty big collection of It's actually a pretty big collection of files. But okay, so the download job has files. But okay, so the download job has files. But okay, so the download job has worked, so now I am able to with my worked, so now I am able to with my worked, so now I am able to with my script script script to easily download to easily download to easily download AI models from Hugging Face AI models from Hugging Face AI models from Hugging Face so that becomes easier to switch. So, so that becomes easier to switch. So, so that becomes easier to switch. So, that is step one done. Then Then we're going to switch to ops and from we're going to switch to ops and from we're going to switch to ops and from split.

And then And then run book run book run book It has prepared a run book for me. Run this run book against each name Run this run book against each name space. space. space. Before merging the PR, it switches Before merging the PR, it switches Before merging the PR, it switches deployment manifest to the new and from deployment manifest to the new and from deployment manifest to the new and from sources. sources. sources. Okay. So, it has created a So, it has created a script script script that is going to migrate the current that is going to migrate the current that is going to migrate the current config maps and secrets. config maps and secrets. config maps and secrets. Because what I've done in this PR is I Because what I've done in this PR is I Because what I've done in this PR is I refactored my code base and deployments refactored my code base and deployments refactored my code base and deployments to to to um use um use um use environment variables rather than environment variables rather than environment variables rather than hard-coded features. hard-coded features. hard-coded features. However However However the the the Like I'm still in the Like I'm still in the Like I'm still in the POC phase. POC phase. POC phase. So secrets management is still being done secrets management is still being done by me directly.

by me directly. by me directly. That is of course not how you would do That is of course not how you would do That is of course not how you would do it in actual production. it in actual production. it in actual production. But I'm slowly But I'm slowly But I'm slowly getting it more and more production getting it more and more production getting it more and more production ready. And actually I'm just going to ready. And actually I'm just going to ready. And actually I'm just going to ask ask ask um Okay, so Uh please store the full merging Uh please store the full merging sequence sequence sequence to disk on the master branch. to disk on the master branch. to disk on the master branch. This This This every this plan every this plan every this plan that you just wrote in line. that you just wrote in line. that you just wrote in line. Next, I want you to run the run books Next, I want you to run the run books Next, I want you to run the run books before before before I merge. Are you enjoying the usage of your Are you enjoying the usage of your Framework laptop? Yes, I'm enjoying that

Framework laptop? Yes, I'm enjoying that Framework laptop? Yes, I'm enjoying that very much. I'm current This is streaming from my I'm current This is streaming from my Framework desktop. But, I am But, I am when I go on the road with my camper van when I go on the road with my camper van when I go on the road with my camper van next month, then I'll be next month, then I'll be next month, then I'll be uh using my Framework laptop and also uh using my Framework laptop and also uh using my Framework laptop and also going to be streaming from there then.

Hm, it set my category to science and Hm, it set my category to science and technology. technology. technology. Is that correct? I thought software engineering was a I thought software engineering was a category on Twitch. Okay, that's weird. I thought I thought Okay, that's weird. I thought I thought software engineering was a category on software engineering was a category on software engineering was a category on Twitch. Twitch. Twitch. Like I have I I know so little about Like I have I I know so little about Like I have I I know so little about Twitch. Twitch. Twitch. Like I'm Like I'm Like I'm browse. browse. browse. The cat categories. So so it's just The cat categories. So so it's just The cat categories. So so it's just chatting. Software and game development, chatting. Software and game development, chatting. Software and game development, right?

right? right? So why when I switch here and I search So why when I switch here and I search So why when I switch here and I search for software for software for software and Oh, here it is. Software and game Oh, here it is. Software and game development. Maybe I should development. Maybe I should development. Maybe I should switch to that. Let's see if I can switch to that. Let's see if I can switch to that. Let's see if I can like switch that in line. like switch that in line. like switch that in line. And did that change my YouTube And did that change my YouTube And did that change my YouTube category? category? category? That should not change. That should be That should not change. That should be That should not change. That should be science and technology. So did that change my category? So did that change my category? Yeah, now I'm at software engineering. Yeah, now I'm at software engineering. Yeah, now I'm at software engineering. Okay, so Okay, so Okay, so now I know that works. I'm also I'm like as we go along, I'm I'm also I'm like as we go along, I'm like figuring this whole streaming thing like figuring this whole streaming thing like figuring this whole streaming thing out uh as as I'm going along. out uh as as I'm going along. out uh as as I'm going along. Okay, so Ah, that's helpful. I now have my Ah, that's helpful. I now have my sequence.

This is a new development in my This is a new development in my workflow. workflow. workflow. I I It's really nice. I I It's really nice. I I It's really nice. So I have a standing desk and So I have a standing desk and So I have a standing desk and I got a treadmill below my desk. I got a treadmill below my desk. I got a treadmill below my desk. And I am I I I'm I'm in a fat loss phase, so I'm I I I'm I'm in a fat loss phase, so I'm trying to lose fat as much as possible trying to lose fat as much as possible trying to lose fat as much as possible and I'm doing pretty well. and I'm doing pretty well. and I'm doing pretty well. I have get I've lost 9 kilos over the I have get I've lost 9 kilos over the I have get I've lost 9 kilos over the past past past what is it? Four five months? what is it? Four five months? what is it? Four five months? But, I'm now at a level where I need to But, I'm now at a level where I need to But, I'm now at a level where I need to do like everything I can and this is a do like everything I can and this is a do like everything I can and this is a pretty easy way to just burn calories. pretty easy way to just burn calories. pretty easy way to just burn calories. So I'm basically just walking all day So I'm basically just walking all day So I'm basically just walking all day and it's actually really it's helping and it's actually really it's helping and it's actually really it's helping with my back. It's with my back. It's with my back. It's good for my I'm burning more calories good for my I'm burning more calories good for my I'm burning more calories and in general I love walking. I love and in general I love walking. I love and in general I love walking. I love going outside going outside going outside um and walk, so I might as well just do um and walk, so I might as well just do um and walk, so I might as well just do it while I work. it while I work. it while I work. Okay.

Uh and from split. So here we see what Uh and from split. So here we see what we're actually implementing. we're actually implementing. we're actually implementing. This was the run book. This was the run book. This was the run book. But, so But, so But, so we had all of these ugly inline we had all of these ugly inline we had all of these ugly inline environment configurations in the environment configurations in the environment configurations in the deployment manifests. deployment manifests. deployment manifests. And that is not how I want to deploy And that is not how I want to deploy And that is not how I want to deploy my applications. my applications. my applications. That is not how you should be running a That is not how you should be running a That is not how you should be running a production grade um production grade um production grade um deployment. deployment. deployment. This is one of the things I teach to my This is one of the things I teach to my This is one of the things I teach to my students that students that students that you should always be using and from and you should always be using and from and you should always be using and from and then you refer to a config map then you refer to a config map then you refer to a config map or a secret. or a secret. or a secret. And then all of your config etc. neatly And then all of your config etc. neatly And then all of your config etc. neatly lives in your secrets. lives in your secrets. lives in your secrets. This now has a merge conflict. This now has a merge conflict. This now has a merge conflict. Let's see. Let's just let the AI handle it. There Let's just let the AI handle it. There is a merge.

Actually, yeah, Linus Torvalds also does Actually, yeah, Linus Torvalds also does this. If you've seen this video of uh or Linus If you've seen this video of uh or Linus Torvalds' desk. Yeah, here. This is his This is Linus Yeah, here. This is his This is Linus Torvalds' workstation. Torvalds' workstation. Torvalds' workstation. This is him on a treadmill and a This is him on a treadmill and a This is him on a treadmill and a small standing desk. small standing desk. small standing desk. And I basically my sim my setup is And I basically my sim my setup is And I basically my sim my setup is actually very similar. actually very similar. actually very similar. I don't think I can detach my camera I don't think I can detach my camera I don't think I can detach my camera very easily, but I I also have one very easily, but I I also have one very easily, but I I also have one street screen, street screen, street screen, just my keyboard just my keyboard just my keyboard and a mouse and that's it. and a mouse and that's it. and a mouse and that's it. One screen. Finished working on my K3S cluster. Finished working on my K3S cluster. Nice.

Nice. Nice. Good job. Okay, merge it. Okay, merge it. Merge it and do the post merge sequence. Merge it and do the post merge sequence. Merge it and do the post merge sequence. And in the meantime, I will get myself a And in the meantime, I will get myself a And in the meantime, I will get myself a drink. All right, so we are All right, so we are rolling out a couple things.

rolling out a couple things. rolling out a couple things. So let's see what's going on. 21 seconds, it already 21 seconds, it already did it. did it. did it. So it has redeployed the back end and So it has redeployed the back end and So it has redeployed the back end and the the the the front end. it is now smoke testing it, but I should it is now smoke testing it, but I should do a test myself. So, let's refresh do a test myself. So, let's refresh do a test myself. So, let's refresh here. Yep. Yep. Still looking good.

Is this local AI? Is this local AI? Well, the the the the application that you see here, this the application that you see here, this the application that you see here, this is running on local AI. is running on local AI. is running on local AI. This is running in my own Kubernetes This is running in my own Kubernetes This is running in my own Kubernetes cluster on a 4090 GPU. cluster on a 4090 GPU. cluster on a 4090 GPU. But, this cloud code is using cloud code But, this cloud code is using cloud code But, this cloud code is using cloud code Opus. So, this is not local AI. Do we switch to atomic system or should Do we switch to atomic system or should we use the Arch Linux? we use the Arch Linux? we use the Arch Linux? Like the idea of atomic system is Like the idea of atomic system is Like the idea of atomic system is robust. Yeah, well, so I really like the Yeah, well, so I really like the read-only file system. read-only file system. read-only file system. It's It's It's It is working really well for me.

It makes me a better engineer, and I It makes me a better engineer, and I actually did a full video on this. actually did a full video on this. actually did a full video on this. Go to my channel. And And Oh guys, look, we just hit 94k subs. Oh guys, look, we just hit 94k subs. Oh guys, look, we just hit 94k subs. That's cool. That's cool. That's cool. Almost 100k. That's really cool. That's really cool. It's shareable link. And how does It's shareable link. And how does It's shareable link. And how does restream chat work? Can I even restream chat work? Can I even restream chat work? Can I even Yes, here I I can chat here. Yes, here I I can chat here. Yes, here I I can chat here. So, So, So, here. here. here. Sharing the YouTube video Sharing the YouTube video Sharing the YouTube video chat. chat. chat. And And And in this video, I basically share all of in this video, I basically share all of in this video, I basically share all of my thoughts on my thoughts on my thoughts on on the on the on the on my switch from from Fedora to from on my switch from from Fedora to from on my switch from from Fedora to from Arch Linux to Fedora. What my thoughts Arch Linux to Fedora. What my thoughts Arch Linux to Fedora. What my thoughts are, are, are, what I like it, why I like it. So, I what I like it, why I like it. So, I what I like it, why I like it. So, I recommend you watch that video if you recommend you watch that video if you recommend you watch that video if you want to have a deep answer.

want to have a deep answer. want to have a deep answer. Great, so this is all working. And let's just review the PR before I And let's just review the PR before I give it the green light. So, that's 135. So, here we are refactoring So, here we are refactoring the code the code the code to use environment variables to use environment variables to use environment variables to configure to configure to configure the model ID. So, this is the actual the model ID. So, this is the actual the model ID. So, this is the actual backend code that we are backend code that we are backend code that we are updating updating updating here. here. here. So, there is a new config.py. So, there is a new config.py. So, there is a new config.py. Yeah, I like this.

Okay. Okay. We have some new tests. When working with Docker and Podman, we When working with Docker and Podman, we should use Fedora. Is that true? Because should use Fedora. Is that true? Because should use Fedora. Is that true? Because in that says that Fedora is giving more in that says that Fedora is giving more in that says that Fedora is giving more support. No, I don't think that's true. support. No, I don't think that's true. support. No, I don't think that's true. You can use any any anything you want. Which model are you using? I'm currently Which model are you using? I'm currently using uh using uh using uh Gemma 4 26B.

Gemma 4 26B. Gemma 4 26B. That's in the production That's in the production That's in the production application. That's the one that's application. That's the one that's application. That's the one that's online right now. And I was trying Yesterday, I was And I was trying Yesterday, I was trying to get Qwen trying to get Qwen trying to get Qwen to run. to run. to run. So, Qwen So, Qwen So, Qwen 3.6. 3.6. 3.6. This is the one I really want to try. And I was going for the 35 billion And I was going for the 35 billion parameters model, but that one I didn't parameters model, but that one I didn't parameters model, but that one I didn't get to work. Then I tried the 27, and get to work. Then I tried the 27, and get to work. Then I tried the 27, and that also didn't work. that also didn't work. that also didn't work. And And And the The thing is that The thing is that I was watching videos of a guy who was I was watching videos of a guy who was I was watching videos of a guy who was running them on running them on running them on using llama.cpp using llama.cpp using llama.cpp C++, C++, C++, and he could run and he could run and he could run the same model on a 4090. the same model on a 4090. the same model on a 4090. And it worked great for him. And it worked great for him. And it worked great for him. But, when I'm doing it, But, when I'm doing it, But, when I'm doing it, I'm using vLLM, I'm using vLLM, I'm using vLLM, and I was 200 megabytes of memory short, and I was 200 megabytes of memory short, and I was 200 megabytes of memory short, unfortunately.

unfortunately. unfortunately. So, I couldn't get it to run. So, I couldn't get it to run. So, I couldn't get it to run. So, what I'm So, what I'm So, what I'm going to What I'm planning to do, going to What I'm planning to do, going to What I'm planning to do, actually I should create an issue for actually I should create an issue for actually I should create an issue for this. this. this. Um Um Um Let's open Claude, and I have a Let's open Claude, and I have a Let's open Claude, and I have a instance that creates issues. >> Create an issue >> Create an issue to to to set up set up set up local AI. So, a a local development setup that AI. So, a a local development setup that does not involve the Kubernetes cluster does not involve the Kubernetes cluster does not involve the Kubernetes cluster at all. at all. at all. So, that I can test larger models. So, that I can test larger models. So, that I can test larger models. Because the Kubernetes cluster only has Because the Kubernetes cluster only has Because the Kubernetes cluster only has a 4090 with 24 gigs of a 4090 with 24 gigs of a 4090 with 24 gigs of VRAM. VRAM. VRAM. But, my Framework desktop has 128 But, my Framework desktop has 128 But, my Framework desktop has 128 gigabytes available.

gigabytes available. gigabytes available. So, we need to So, we need to So, we need to create a setup so that I can spin the create a setup so that I can spin the create a setup so that I can spin the entire application up with Podman entire application up with Podman entire application up with Podman using a Podman compose file. using a Podman compose file. using a Podman compose file. So, I can test a staging version of the So, I can test a staging version of the So, I can test a staging version of the application application application locally while running the the models locally. while running the the models locally. However, it is completely fine to be However, it is completely fine to be However, it is completely fine to be talking to the the talking to the the talking to the the the same to the Superbase instance. So, the same to the Superbase instance. So, the same to the Superbase instance. So, we don't need to be we don't need to be we don't need to be We don't need a separate Superbase for We don't need a separate Superbase for We don't need a separate Superbase for this. this. this. The It's all about The It's all about The It's all about the the the the local AI model testing. Podman or Docker, which is best?

Podman or Docker, which is best? I like Podman. I've used Docker like my entire career, I've used Docker like my entire career, so there's nothing wrong with Docker. so there's nothing wrong with Docker. so there's nothing wrong with Docker. But, Podman is open source. It is But, Podman is open source. It is But, Podman is open source. It is maintained by Red Hat maintained by Red Hat maintained by Red Hat rather than Docker. And Docker is really rather than Docker. And Docker is really rather than Docker. And Docker is really like trying to find its identity as a like trying to find its identity as a like trying to find its identity as a company. company. company. So, I personally lean towards Podman. So, I personally lean towards Podman. So, I personally lean towards Podman. Uh let's see. What do we have now? Uh let's see. What do we have now? Uh let's see. What do we have now? Pre-merge patched. Okay. Pre-merge patched. Okay. Pre-merge patched. Okay. Staging bump merged. Staging bump merged. Staging bump merged. Staging is healthy. Let's see. Let's see. Yep, let's go. Then, this should then trigger a Then, this should then trigger a redeploy of production. redeploy of production. redeploy of production. Which we should see here.

>> Okay, now it's sinking the >> Okay, now it's sinking the customization. Let's Let's test the actual application again. Yep. Yep. That's working. Next step, Next step, no pre-merge.

Yep, let's do it. Yep, let's do it. But let me just check the PR first. But let me just check the PR first. But let me just check the PR first. Just to see what we're actually Just to see what we're actually Just to see what we're actually touching. So this one was 135. So this one was 135. Or what was it? Or what was it? Or what was it? 36. Yes, so here we are also refactoring Yes, so here we are also refactoring everything into environment variables. everything into environment variables. everything into environment variables. Okay. What do you think about Origami Linux? I What do you think about Origami Linux? I heard it's basically Fedora Atomic with heard it's basically Fedora Atomic with heard it's basically Fedora Atomic with a catchy OS kernel. a catchy OS kernel. a catchy OS kernel. Do you think the optimizations are worth Do you think the optimizations are worth Do you think the optimizations are worth it? Actually on Catch your OS. it? Actually on Catch your OS. it? Actually on Catch your OS. Like one of my students, he really he Like one of my students, he really he Like one of my students, he really he shared a very thoughtful post shared a very thoughtful post shared a very thoughtful post um And by the way, this is how active the And by the way, this is how active the community is, right? Like it's so good.

community is, right? Like it's so good. community is, right? Like it's so good. No, this is not it. Where is the Where is the Oleksandr, you were the one who Oleksandr, you were the one who Oleksandr, you were the one who Here. Why I decided to leave Catch your Here. Why I decided to leave Catch your Here. Why I decided to leave Catch your OS. OS. OS. So the co-founder and lead architect of So the co-founder and lead architect of So the co-founder and lead architect of Catch your OS resides in the Russian Catch your OS resides in the Russian Catch your OS resides in the Russian Federation. Federation. Federation. Those cannot remain a political Those cannot remain a political Those cannot remain a political um Catch your OS uses experimental untested Catch your OS uses experimental untested patches and custom kernels to squeeze a patches and custom kernels to squeeze a patches and custom kernels to squeeze a tiny performance boost in games. tiny performance boost in games. tiny performance boost in games. And it has security concerns. So like And it has security concerns. So like And it has security concerns. So like this he this man is very thoughtful and this he this man is very thoughtful and this he this man is very thoughtful and he puts a lot of he puts a lot of he puts a lot of thoughts in into these kinds of thoughts in into these kinds of thoughts in into these kinds of decisions. decisions. decisions. That's the kind of people we have in That's the kind of people we have in That's the kind of people we have in CubeCraft. CubeCraft. CubeCraft. And And And it's um he he decided to leave it. So I it's um he he decided to leave it. So I it's um he he decided to leave it. So I I'm actually not considering Catch your I'm actually not considering Catch your I'm actually not considering Catch your OS at all. OS at all. OS at all. And by extension then also not Origami And by extension then also not Origami And by extension then also not Origami Linux.

Origami Origami Linux. Linux. Linux. Fold your system. Yeah, I mean Yeah, I mean if you want Fedora Atomic with Cosmic if you want Fedora Atomic with Cosmic if you want Fedora Atomic with Cosmic Desktop's elegance, like why not just Desktop's elegance, like why not just Desktop's elegance, like why not just get Fedora Atomic Cosmic? Like I don't understand why Like I don't understand why something different needs to be built if something different needs to be built if something different needs to be built if you want Fedora Atomic and Cosmic. you want Fedora Atomic and Cosmic. you want Fedora Atomic and Cosmic. Keep things simple, people. >> Oh, by the way, guys, in the description >> Oh, by the way, guys, in the description of the of the of the Actually, is that even showing? Let's Actually, is that even showing? Let's Actually, is that even showing? Let's check.

check. check. Building AI apps on local AI and Building AI apps on local AI and Building AI apps on local AI and Kubernetes. Kubernetes. Kubernetes. Cool. Cool. Cool. Yeah. So in the description, there is a Yeah. So in the description, there is a Yeah. So in the description, there is a link here and I'm also going to share it link here and I'm also going to share it link here and I'm also going to share it in the chat. in the chat. in the chat. So my first DevOps community is has been So my first DevOps community is has been So my first DevOps community is has been a massive success and people we've been a massive success and people we've been a massive success and people we've been running it for over two years. running it for over two years. running it for over two years. We're landing jobs every single week. We're landing jobs every single week. We're landing jobs every single week. And it's going really well. So And it's going really well. So And it's going really well. So I am considering to start a second I am considering to start a second I am considering to start a second community community community for AI engineering because this is for AI engineering because this is for AI engineering because this is basically what I've been doing for the basically what I've been doing for the basically what I've been doing for the past past past uh year or so. I've been uh year or so. I've been uh year or so. I've been learning AI learning AI learning AI and getting even deeper into Kubernetes, and getting even deeper into Kubernetes, and getting even deeper into Kubernetes, of course. of course. of course. And I now help people land DevOps jobs. And I now help people land DevOps jobs. And I now help people land DevOps jobs. So that is my my main focus. So that is my my main focus. So that is my my main focus. But for them, I'm developing these AI But for them, I'm developing these AI But for them, I'm developing these AI applications and applications and applications and my personal career trajectory is now my personal career trajectory is now my personal career trajectory is now heading towards ML Ops and AI heading towards ML Ops and AI heading towards ML Ops and AI infrastructure engineering. So that's infrastructure engineering. So that's infrastructure engineering. So that's why I'm building this these why I'm building this these why I'm building this these applications, too.

applications, too. applications, too. And And And I know that there are a lot of people in I know that there are a lot of people in I know that there are a lot of people in my audience who do not my audience who do not my audience who do not uh they do not need to become a DevOps uh they do not need to become a DevOps uh they do not need to become a DevOps engineer or they're not interested in engineer or they're not interested in engineer or they're not interested in that. that. that. They are but like I know for a fact that They are but like I know for a fact that They are but like I know for a fact that there are many people who are interested there are many people who are interested there are many people who are interested in in in um um um all of the AI stuff that I'm doing. So I all of the AI stuff that I'm doing. So I all of the AI stuff that I'm doing. So I have have have like I'm running a company like I'm running a company like I'm running a company a multi-six-figure company with just me a multi-six-figure company with just me a multi-six-figure company with just me and my brother and my brother and my brother and and one other person in on the and and one other person in on the and and one other person in on the operations. operations. operations. And so like five years ago, that would And so like five years ago, that would And so like five years ago, that would have been completely impossible with the have been completely impossible with the have been completely impossible with the amount of content we create and the amount of content we create and the amount of content we create and the amount of support we give to our amount of support we give to our amount of support we give to our students. students. students. Uh so like we are a very AI-first Uh so like we are a very AI-first Uh so like we are a very AI-first company. company. company. And And And I know that there are many people in my I know that there are many people in my I know that there are many people in my audience interested in my specific AI audience interested in my specific AI audience interested in my specific AI workflows, how I do research, how I you workflows, how I do research, how I you workflows, how I do research, how I you even create content with AI sometimes. even create content with AI sometimes. even create content with AI sometimes. So So So for everyone who is interested in for everyone who is interested in for everyone who is interested in learning more about AI from me and how I learning more about AI from me and how I learning more about AI from me and how I use AI, how I'm building these use AI, how I'm building these use AI, how I'm building these applications applications applications I am I am I am considering to launch a new AI considering to launch a new AI considering to launch a new AI community.

community. community. And And And if you want to join the waitlist for if you want to join the waitlist for if you want to join the waitlist for that, you can go to the link in the that, you can go to the link in the that, you can go to the link in the chat. chat. chat. You can sign up for the waitlist. Like You can sign up for the waitlist. Like You can sign up for the waitlist. Like it's not completely sure if I'm even it's not completely sure if I'm even it's not completely sure if I'm even going to do it. going to do it. going to do it. Uh it depends on what people are Uh it depends on what people are Uh it depends on what people are interested in, etc. interested in, etc. interested in, etc. But if we do it, then if you sign up for But if we do it, then if you sign up for But if we do it, then if you sign up for the waitlist now, then you're going to the waitlist now, then you're going to the waitlist now, then you're going to become a founding member and you're become a founding member and you're become a founding member and you're going to get a big discount on the going to get a big discount on the going to get a big discount on the price, which you will lock in forever. price, which you will lock in forever. price, which you will lock in forever. So So So if you are at all interested in learning if you are at all interested in learning if you are at all interested in learning AI um from me at some point then AI um from me at some point then AI um from me at some point then definitely join the waitlist because you definitely join the waitlist because you definitely join the waitlist because you will lock in for a very cheap price. will lock in for a very cheap price. will lock in for a very cheap price. And uh it will it And uh it will it And uh it will it you're grandfathered in at that price, you're grandfathered in at that price, you're grandfathered in at that price, too. too. too. The the waitlist is completely for free The the waitlist is completely for free The the waitlist is completely for free and you can um and you can um and you can um check it out. check it out. check it out. It's completely free. There's no no no It's completely free. There's no no no It's completely free. There's no no no strings attached, but I'm just trying to strings attached, but I'm just trying to strings attached, but I'm just trying to gather gather gather um the how much interest there is for this, the how much interest there is for this, right? So if I get right? So if I get right? So if I get a few hundred people to sign up for the a few hundred people to sign up for the a few hundred people to sign up for the waitlist, then I know, okay waitlist, then I know, okay waitlist, then I know, okay people are interested in this and I people are interested in this and I people are interested in this and I should do this.

So if you're interested in learning AI So if you're interested in learning AI from me from me from me or joining a AI community that's or joining a AI community that's or joining a AI community that's centered around my sort of approach centered around my sort of approach centered around my sort of approach being an AI craftsman, then um being an AI craftsman, then um being an AI craftsman, then um check out the free waitlist. It's in the check out the free waitlist. It's in the check out the free waitlist. It's in the in the chat. in the chat. in the chat. All right, prediction production chat UI All right, prediction production chat UI All right, prediction production chat UI on the new image. on the new image. on the new image. So that is now And everything seems to be working just And everything seems to be working just fine. fine. fine. Very cool. Excellent. So the back end and the chat Excellent. So the back end and the chat UI are now reading their model identity UI are now reading their model identity UI are now reading their model identity entirely from imperative config maps. entirely from imperative config maps. entirely from imperative config maps. A swap going forward is to patch the A swap going forward is to patch the A swap going forward is to patch the config map, roll out restart, no code config map, roll out restart, no code config map, roll out restart, no code changes. Create an issue to deploy reloader for Create an issue to deploy reloader for automatically restarting deployments automatically restarting deployments automatically restarting deployments when config maps change.

Actually, is this also adding a Actually, is this also adding a Yeah, P2. Nice. It's reading the My work is implementing Workday and My work is implementing Workday and Opus, I think. What do you mean? What do you mean? Workday and Opus, I think. Workday and Opus, I think. Workday and Opus, I think. Healthcare supply software. Healthcare supply software. Healthcare supply software. Can you elaborate, Kilobytes? Like, what Can you elaborate, Kilobytes? Like, what Can you elaborate, Kilobytes? Like, what do you do? And what do you mean with I do you do? And what do you mean with I do you do? And what do you mean with I think? think? think? >> [laughter] >> [laughter] >> [laughter] >> Are you not sure what AI you're >> Are you not sure what AI you're >> Are you not sure what AI you're implementing? implementing? implementing? >> [snorts] >> [snorts] >> [snorts] >> Cuz that would be pretty funny. >> Cuz that would be pretty funny. >> Cuz that would be pretty funny. Let's see. Let's see. Let's see. So, we are live. The So, we are live. The So, we are live. The We are now able to change configuration We are now able to change configuration We are now able to change configuration by just swapping the config maps because by just swapping the config maps because by just swapping the config maps because what was happening before was there were what was happening before was there were what was happening before was there were hardcoded values in my code hardcoded values in my code hardcoded values in my code which required a full new deployment of which required a full new deployment of which required a full new deployment of the backend, for example, if I switch the backend, for example, if I switch the backend, for example, if I switch the model, which was very painful. So, the model, which was very painful. So, the model, which was very painful. So, I wanted to have everything controlled I wanted to have everything controlled I wanted to have everything controlled through environment variables, which is through environment variables, which is through environment variables, which is what you should be doing in modern what you should be doing in modern what you should be doing in modern application development. This is from application development. This is from application development. This is from the framework 12-factor app, if you're the framework 12-factor app, if you're the framework 12-factor app, if you're interested.

interested. interested. Last PR, use model script Claude MD Last PR, use model script Claude MD Last PR, use model script Claude MD conventions. Nothing. So, let's just check out what that PR is So, let's just check out what that PR is all about. Use model. Oh, yeah. So, this is now a Use model. Oh, yeah. So, this is now a script that is going to swap a model. Okay. So, I already don't like the fact Okay. So, I already don't like the fact that that that these are hardcoded both these are hardcoded both these are hardcoded both in the download model script and in the Oh, yeah. So, it basically just

Oh, yeah. So, it basically just regenerates the config maps with the regenerates the config maps with the regenerates the config maps with the model ID model ID model ID and then restarts the deployment. Yes. Yes. Let's do it. Add issue. I just spotted [snorts] Add issue. I just spotted [snorts] that we are that we are that we are that we have two places where we are that we have two places where we are that we have two places where we are storing the list of available models. storing the list of available models. storing the list of available models. So, that's both in the download model So, that's both in the download model So, that's both in the download model script and in the use model script.

script and in the use model script. script and in the use model script. I would rather have this I would rather have this I would rather have this handled in perhaps one script handled in perhaps one script handled in perhaps one script or at least it should be a or at least it should be a or at least it should be a At least the the place where we list all At least the the place where we list all At least the the place where we list all of the models should just be one file of the models should just be one file of the models should just be one file and not be done in multiple files. and not be done in multiple files. and not be done in multiple files. And perhaps we are at the stage where we And perhaps we are at the stage where we And perhaps we are at the stage where we should consider should consider should consider refactoring to refactoring to refactoring to a a a full full full Typer CLI Typer CLI Typer CLI where we are going to where we are going to where we are going to basically create a CubeCraft AI CLI for basically create a CubeCraft AI CLI for basically create a CubeCraft AI CLI for management. management. management. And And And the use model, etc., is all going to be the use model, etc., is all going to be the use model, etc., is all going to be implemented in the same implemented in the same implemented in the same CLI rather than having multiple places CLI rather than having multiple places CLI rather than having multiple places and scripts and scripts and scripts so that we have one place where all of so that we have one place where all of so that we have one place where all of this configuration is stored and is this configuration is stored and is this configuration is stored and is happening.

Oh, the enterprise AI platform for Oh, the enterprise AI platform for managing HR and finance. Super intelligence for work. Super intelligence for work. Empower entire your entire organization Empower entire your entire organization Empower entire your entire organization to reimagine work. AI agent set. Yeah, payroll agent. Oh, AI agent set. Yeah, payroll agent. Oh, yeah. yeah. yeah. Interesting. Pretty cool. Interesting. Pretty cool. Interesting. Pretty cool. Yeah, doing implementations of that is Yeah, doing implementations of that is Yeah, doing implementations of that is definitely good experience to have. Yes, file those issues Yes, file those issues with appropriate Next, let's Next, let's Should I test the Maybe we should test with a different Maybe we should test with a different Um

when 3.6 when 3.6 It is actually only available as 27B and It is actually only available as 27B and It is actually only available as 27B and I was not able to get that to run. So, I I really think that the the next So, I I really think that the the next logical step is going to be to be able logical step is going to be to be able logical step is going to be to be able to run these on my actual local system. to run these on my actual local system. to run these on my actual local system. And when I confirm that the application And when I confirm that the application And when I confirm that the application runs well with that model, then I'm runs well with that model, then I'm runs well with that model, then I'm going to switch. going to switch. going to switch. So, I've now built the model switching So, I've now built the model switching So, I've now built the model switching infrastructure infrastructure infrastructure which works. which works. which works. However, like since it's now working online, it's like since it's now working online, it's working pretty decently, working pretty decently, working pretty decently, I think I'll have to I think I'll have to I think I'll have to I'm not going to actually test the I'm not going to actually test the I'm not going to actually test the switching at this stage. That can happen switching at this stage. That can happen switching at this stage. That can happen later. So, next is going to be So, next is going to be to decide what the next thing is that to decide what the next thing is that to decide what the next thing is that we're going to be working on.

Yes, so definitely the next one is Yes, so definitely the next one is um um um no conversation memory. no conversation memory. no conversation memory. That is a based on some feedback that That is a based on some feedback that That is a based on some feedback that I've given I've gotten on the I've given I've gotten on the I've given I've gotten on the application. The issue with that is that this is a The issue with that is that this is a fairly big one to do fairly big one to do fairly big one to do and I'm going to and I'm going to and I'm going to be streaming for maximum 1 more hour be streaming for maximum 1 more hour be streaming for maximum 1 more hour and then I have I have to go and eat and and then I have I have to go and eat and and then I have I have to go and eat and I have a visitor coming this afternoon. I have a visitor coming this afternoon. I have a visitor coming this afternoon. So, So, So, I might stream more this this evening. I might stream more this this evening. I might stream more this this evening. But this is But I'm trying to say is But this is But I'm trying to say is But this is But I'm trying to say is that this is basically a solid that this is basically a solid that this is basically a solid Um I will need like a 3-4 hour block to Um I will need like a 3-4 hour block to Um I will need like a 3-4 hour block to to attack this one. to attack this one. to attack this one. So, perhaps for now, I could fire off So, perhaps for now, I could fire off So, perhaps for now, I could fire off the research and and label label label P2.

I could fire up the research for that I could fire up the research for that one. one. one. Wow, there's a lot of P2 issues here. Wow, there's a lot of P2 issues here. Wow, there's a lot of P2 issues here. Yeah. That's the good thing. Like I've I've That's the good thing. Like I've I've deployed the application deployed the application deployed the application and I onboarded my test users and they and I onboarded my test users and they and I onboarded my test users and they started giving me feedback started giving me feedback started giving me feedback and that is then generating all of these and that is then generating all of these and that is then generating all of these issues. issues. issues. Um so there's like all I'm I'm going to Um so there's like all I'm I'm going to Um so there's like all I'm I'm going to be working on this for weeks, like the be working on this for weeks, like the be working on this for weeks, like the next weeks. So, if you're next weeks. So, if you're next weeks. So, if you're interested in watching me build a interested in watching me build a interested in watching me build a application from scratch application from scratch application from scratch and learn about AI, and learn about AI, and learn about AI, then then then um Hello from the Netherlands. Hello there, Hello from the Netherlands. Hello there, Coen. How are you today? Coen. How are you today? Coen. How are you today? Hi Joti. Hi Joti. Hi Joti. Thanks for tuning in. Thanks for tuning in. Thanks for tuning in. Onesloth Coen 3.6. Yes, actually Onesloth Coen 3.6. Yes, actually Onesloth Coen 3.6. Yes, actually someone actually someone actually someone actually mentioned that.

mentioned that. mentioned that. Um that I should perhaps try Onesloth. that I should perhaps try Onesloth. I have heard the term Onesloth I have heard the term Onesloth I have heard the term Onesloth but I have no idea what Onesloth is. So, So, let's take a look at that. let's take a look at that. let's take a look at that. onesloth.ai What is Onesloth? What is Onesloth? Onesloth is an open-source framework for Onesloth is an open-source framework for Onesloth is an open-source framework for running and training models. For a beginner, it might be the first For a beginner, it might be the first question you ask where you fine-tune. question you ask where you fine-tune. question you ask where you fine-tune. Okay, so this is for fine-tuning and Okay, so this is for fine-tuning and Okay, so this is for fine-tuning and running. running. running. Yeah, very cool. So, I have not used Yeah, very cool. So, I have not used Yeah, very cool. So, I have not used Onesloth Onesloth Onesloth but they also do models apparently. but they also do models apparently. but they also do models apparently. Um this looks really good. Um this looks really good. Um this looks really good. So, I'm going to be So, I'm going to be So, I'm going to be catch capturing this catch capturing this catch capturing this um AI.

And let's see. And let's see. Local AI. Local AI. Local AI. How are we going to put this? How are we going to put this? How are we going to put this? It should probably be in tooling. Onesloth Onesloth Oops. Countered by Coen Countered by Coen 3.6 3.6 3.6 benchmark video benchmark video benchmark video was later by Rio. by Rio. Um Um Um Onesloth Onesloth Onesloth Coen Coen Coen 3.6 might be 3.6 might be 3.6 might be a better fit for the a better fit for the a better fit for the refer AI cluster. refer AI cluster. refer AI cluster. And actually And actually And actually that might be fun to try actually. that might be fun to try actually. that might be fun to try actually. Um Yeah, so I tried to run the 4-bit Yeah, so I tried to run the 4-bit quantization quantization quantization of of of 35B actually if I just

actually if I just give it this give it this give it this page page page perhaps it should be able to figure it perhaps it should be able to figure it perhaps it should be able to figure it out. But maybe if I do 4-bit out. But maybe if I do 4-bit out. But maybe if I do 4-bit quantization and the 27B1 and the 27B1 that might actually work. did I also see? Yes, so Onesloth is on did I also see? Yes, so Onesloth is on Hugging Face. 27B4

27B4 bit Yeah, this is probably it then. Yeah, this is probably it then. This one could be fun to try This one could be fun to try This one could be fun to try when it's done. Which window manager and bar are you Which window manager and bar are you using? I'm using Sway and using Waybar. using? I'm using Sway and using Waybar. using? I'm using Sway and using Waybar. Signed up, cool. Signed up, cool. Signed up, cool. Did you try locally first? Uh no. Did you try locally first? Uh no. Did you try locally first? Uh no. So, I So, I So, I I um I just created an issue that I need to I just created an issue that I need to have a have a have a local version of this but honestly I local version of this but honestly I local version of this but honestly I just went straight for Kubernetes in just went straight for Kubernetes in just went straight for Kubernetes in I'm still in like the POC phase. I'm still in like the POC phase. I'm still in like the POC phase. I now have maybe 30 I now have maybe 30 I now have maybe 30 um um um 30-40 test users. 30-40 test users. 30-40 test users. So, I just went straight stick straight So, I just went straight stick straight So, I just went straight stick straight into it.

into it. into it. But I have now an issue where I'm going But I have now an issue where I'm going But I have now an issue where I'm going to try to pull larger local models to try to pull larger local models to try to pull larger local models because I was actually because I was actually because I was actually um not able to run Coen 3.6. um not able to run Coen 3.6. um not able to run Coen 3.6. But actually this would be fun to try. But actually this would be fun to try. But actually this would be fun to try. Okay, so let's see. let's see. Do I do this in the same Do I do this in the same Do I do this in the same session? We had large issues with deploying We had large issues with deploying our Coen our Coen our Coen 3.6 3.6 3.6 recently.

recently. recently. However, I have a However, I have a However, I have a uh I got a new tip to use the Onesloth I got a new tip to use the Onesloth models and I was actually looking at models and I was actually looking at models and I was actually looking at this page this page this page and it seems that there is a Coen 3.6 and it seems that there is a Coen 3.6 and it seems that there is a Coen 3.6 alternative that will fit our 24 alternative that will fit our 24 alternative that will fit our 24 gigabytes of VRAM. gigabytes of VRAM. gigabytes of VRAM. So, I need you to review the settings So, I need you to review the settings So, I need you to review the settings that we used earlier that we used earlier that we used earlier and then tune it based on the current and then tune it based on the current and then tune it based on the current documentation documentation documentation so that we can try out so that we can try out so that we can try out this particular this particular this particular quantization of the quantization of the quantization of the 3.6. 3.6. 3.6. And next And next And next I also we have just implemented that we I also we have just implemented that we I also we have just implemented that we can switch through environment variables can switch through environment variables can switch through environment variables instead. So, this could be a good way to instead. So, this could be a good way to instead. So, this could be a good way to a good use case to test it with the new a good use case to test it with the new a good use case to test it with the new scripting architecture.

Hugging Face Hugging Face and then the documentation. Rio and me. Yes, you're right, Rio and me. Yes, you're right, Sebastian. I will also say plus Sebastian I will also say plus Sebastian on on on YouTube [clears throat] YouTube [clears throat] YouTube [clears throat] stream. stream. stream. Sebastian, you are now officially a part Sebastian, you are now officially a part Sebastian, you are now officially a part of my Zettelkasten, my note-taking of my Zettelkasten, my note-taking of my Zettelkasten, my note-taking system. system. system. Enjoy it. Enjoy it. Enjoy it. What walking pad are you using and do What walking pad are you using and do What walking pad are you using and do you recommend? I literally have no idea. The The the brand says VirtuFit. the brand says VirtuFit. the brand says VirtuFit. Let me see if I can share that. I um I um I my brother, he bought this a while ago I my brother, he bought this a while ago I my brother, he bought this a while ago and then he and then he and then he uh I basically uh I basically uh I basically gave it he gave it to me gave it he gave it to me gave it he gave it to me so I'm using I just started using that so I'm using I just started using that so I'm using I just started using that so I don't really have any so I don't really have any so I don't really have any recommendations.

recommendations. recommendations. I like this one. What I would have like liked is that it What I would have like liked is that it it if it would sync my steps to my it if it would sync my steps to my it if it would sync my steps to my iPhone iPhone iPhone because I'm I track you know my steps because I'm I track you know my steps because I'm I track you know my steps with whoop and everything with whoop and everything with whoop and everything but but but because I'm having my hands on the because I'm having my hands on the because I'm having my hands on the keyboard my whoop is not registering my steps so my registering my steps so my caloric like the calories I burn is caloric like the calories I burn is caloric like the calories I burn is actually not being registered at all which is not being registered at all which is annoying. annoying. annoying. Yes, it's a whoop. Hyperland scares me. I end up looking at Hyperland scares me. I end up looking at the config files the entire time. Yeah. the config files the entire time. Yeah. the config files the entire time. Yeah. Yeah, I was on Hyperland for well over a Yeah, I was on Hyperland for well over a Yeah, I was on Hyperland for well over a year but I've now finally come home on year but I've now finally come home on year but I've now finally come home on sway. Sway is the way to go. So nice. sway. Sway is the way to go. So nice. sway. Sway is the way to go. So nice. Seems like you don't have a local Seems like you don't have a local Seems like you don't have a local development workflow where you test development workflow where you test development workflow where you test things local first. No, that is actually things local first. No, that is actually things local first. No, that is actually one of the next things that I'm going to one of the next things that I'm going to one of the next things that I'm going to be working on be working on be working on but but but I basically I basically I basically went straight into deploying right to went straight into deploying right to went straight into deploying right to Kubernetes Kubernetes Kubernetes so when I say local like why Kubernetes so when I say local like why Kubernetes so when I say local like why Kubernetes if you go for local use only what I mean if you go for local use only what I mean if you go for local use only what I mean when I say local is that I'm actually when I say local is that I'm actually when I say local is that I'm actually running this these models on my running this these models on my running this these models on my Kubernetes cluster in a GPU.

Kubernetes cluster in a GPU. Kubernetes cluster in a GPU. So So So I am I am I am running this locally on a Kubernetes running this locally on a Kubernetes running this locally on a Kubernetes cluster cluster cluster so it is local AI. so it is local AI. so it is local AI. The actual application that is that is The actual application that is that is The actual application that is that is local local local but I I still have to set up as you say but I I still have to set up as you say but I I still have to set up as you say a proper development flow where I'm running the development flow where I'm running the model locally on my desktop here. model locally on my desktop here. model locally on my desktop here. Destiny application there and then Destiny application there and then Destiny application there and then promoting it to Kubernetes. I I promoting it to Kubernetes. I I promoting it to Kubernetes. I I literally haven't uh literally haven't uh literally haven't uh come that far yet. come that far yet. come that far yet. I just wanted to have a I just wanted to have a I just wanted to have a good new I just wanted to have the first good new I just wanted to have the first good new I just wanted to have the first version of the application online to get version of the application online to get version of the application online to get my test users which has worked really my test users which has worked really my test users which has worked really well. The application is running well. The application is running well. The application is running amazingly well on Kubernetes locally on amazingly well on Kubernetes locally on amazingly well on Kubernetes locally on my GPU. So it works great. Results are my GPU. So it works great. Results are my GPU. So it works great. Results are good enough and now I'm just you know good enough and now I'm just you know good enough and now I'm just you know improving as I go.

The Onslaught are GGUF only. The Onslaught are GGUF only. To run them To run them To run them we have to go away from vllm we have to go away from vllm we have to go away from vllm and use llama.cpp. Okay, Okay, I don't like that I don't like that I don't like that because KServe Let's Let's see. Here we have yarn layer. Yeah, so KServe Here we have yarn layer. Yeah, so KServe is is is basically vllm only. basically vllm only. basically vllm only. And I could probably just I could switch it I could switch it but I want to stick to what KServe is doing. KServe is doing. So I don't want to So I don't want to So I don't want to Let me see.

Let me see. Let me see. architecture I don't want to deviate from what KServe I don't want to deviate from what KServe is doing. Does this Does this have the docs in here too? KServe KServe website website website Okay. Let's clone the website. Let's clone the website. Clone this repo and study the docs. Clone this repo and study the docs. Clone this repo and study the docs. They They They are do not are do not are do not KServe is using vllm KServe is using vllm KServe is using vllm and I don't wish to deviate from that and I don't wish to deviate from that and I don't wish to deviate from that from their from their from their architecture.

architecture. architecture. Do they Do they Do they even mention llama even mention llama even mention llama cpp cpp cpp or anything or anything or anything to make this viable? Onslaught is the MLX variant Apple Onslaught is the MLX variant Apple silicon only. But why is it that vllm cannot run this? But why is it that vllm cannot run this? So let's block. 4090 4090 and I'm My assistant My assistant is telling me

My stepper has a small display to show My stepper has a small display to show calories and steps. Yes, it does have calories and steps. Yes, it does have calories and steps. Yes, it does have that that that as well. as well. as well. So I'm I'm able to see the steps So I'm I'm able to see the steps So I'm I'm able to see the steps but I want that to I want that to sync to my iPhone, right? So um um Like I cannot like enter my step count Like I cannot like enter my step count Like I cannot like enter my step count somewhere as far as I know. Are planning to use K3D when setting up Are planning to use K3D when setting up local development workflow? Um in this local development workflow? Um in this local development workflow? Um in this case probably I will case probably I will case probably I will not run the Kubernetes cluster locally. not run the Kubernetes cluster locally. not run the Kubernetes cluster locally. I would just go and I would just go and I would just go and run I would just spin up the application run I would just spin up the application run I would just spin up the application in a container in a container in a container and have a model served from a container and have a model served from a container and have a model served from a container and test it that way.

Really enjoy your stream where you Really enjoy your stream where you explain the way you think as a explain the way you think as a explain the way you think as a Kubernetes expert. Kubernetes expert. Kubernetes expert. As a VI engineer this learns me a lot. As a VI engineer this learns me a lot. As a VI engineer this learns me a lot. Cool. Yeah, I mean Cool. Yeah, I mean Cool. Yeah, I mean Thanks Like I'm I'm um Thanks Like I'm I'm um Thanks Like I'm I'm um Like I find myself on this intersection Like I find myself on this intersection Like I find myself on this intersection of Kubernetes and AI of Kubernetes and AI of Kubernetes and AI which is a really interesting place to which is a really interesting place to which is a really interesting place to be because I've been using AI like every be because I've been using AI like every be because I've been using AI like every day day day for 8 plus hours for 8 plus hours for 8 plus hours um for um for um for over 2 years now. over 2 years now. over 2 years now. And I'm I'm actually getting really good And I'm I'm actually getting really good And I'm I'm actually getting really good at using these tools and now I'm at using these tools and now I'm at using these tools and now I'm actually actually actually um um um building my own applications using AI. building my own applications using AI. building my own applications using AI. So I have this Kubernetes expertise that So I have this Kubernetes expertise that So I have this Kubernetes expertise that is now coming together really nicely is now coming together really nicely is now coming together really nicely with this. with this. with this. Um so if you saw that I'm I am going to Um so if you saw that I'm I am going to Um so if you saw that I'm I am going to create an AI community create an AI community create an AI community I will share the link with you. I will share the link with you. I will share the link with you. um um um Like I'm I'm not sure if I'm going to do Like I'm I'm not sure if I'm going to do Like I'm I'm not sure if I'm going to do it but if enough people are interested I it but if enough people are interested I it but if enough people are interested I will. will. will. And I will be And I will be And I will be um building a community. You can sign up um building a community. You can sign up um building a community. You can sign up for the waitlist. And like since you're for the waitlist. And like since you're for the waitlist. And like since you're saying that this helps you like I'm saying that this helps you like I'm saying that this helps you like I'm going to be streaming a lot more in the going to be streaming a lot more in the going to be streaming a lot more in the future building this kind of stuff.

future building this kind of stuff. future building this kind of stuff. But yeah, you may consider signing up But yeah, you may consider signing up But yeah, you may consider signing up there because then we can connect even there because then we can connect even there because then we can connect even further in the future. further in the future. further in the future. Um why did you ask Grok and not Claude? Um why did you ask Grok and not Claude? Um why did you ask Grok and not Claude? Well Well Well the the the there are a few reasons for this. there are a few reasons for this. there are a few reasons for this. I really like to use Grok for research I really like to use Grok for research I really like to use Grok for research because Grok has access to X the X because Grok has access to X the X because Grok has access to X the X platform platform platform and like in this case it I I didn't need and like in this case it I I didn't need and like in this case it I I didn't need to, to, to, but it will but it will but it will like it looked it used three 230 like it looked it used three 230 like it looked it used three 230 sources, right? So it browsed the web. sources, right? So it browsed the web. sources, right? So it browsed the web. It it automatically spawns multiple It it automatically spawns multiple It it automatically spawns multiple agents when doing research. So agents when doing research. So agents when doing research. So that it that works that it that works that it that works really well for research in my um um um what do you call it? Experience. And what do you call it? Experience. And what do you call it? Experience. And especially when I want to have like the especially when I want to have like the especially when I want to have like the latest and greatest on AI you can tell latest and greatest on AI you can tell latest and greatest on AI you can tell it to search X. It works super well. it to search X. It works super well. it to search X. It works super well. So So So for like asking questions I usually for like asking questions I usually for like asking questions I usually default to Grok.

default to Grok. default to Grok. And because I have a paid X account I And because I have a paid X account I And because I have a paid X account I you basically get a Grok for free. you basically get a Grok for free. you basically get a Grok for free. And it's also a way to save some tokens, And it's also a way to save some tokens, And it's also a way to save some tokens, right? If I can get the answer from Grok right? If I can get the answer from Grok right? If I can get the answer from Grok uh if I don't get a good answer then uh if I don't get a good answer then uh if I don't get a good answer then I'll just go back to Claude. I'll just go back to Claude. I'll just go back to Claude. Okay, the short answer Okay, the short answer Okay, the short answer the onslaught model and its variants is the onslaught model and its variants is the onslaught model and its variants is GGML have quantized GGML have quantized GGML have quantized specifically built for Llama CPP. specifically built for Llama CPP. specifically built for Llama CPP. VLM cannot run this. Okay, so that is VLM cannot run this. Okay, so that is VLM cannot run this. Okay, so that is interesting. interesting. interesting. Um I had some notes on Um I had some notes on Um I had some notes on uh Um GGML Um GGML quantization is not quantization is not quantization is not compatible Actually does that Actually does that >> [sighs and gasps] >> [sighs and gasps] >> [sighs and gasps] >> No, it wasn't even here. So

So GGML is it So GGML is it So what is GGML? So So I'm I'm familiar with like four-bit I'm I'm familiar with like four-bit I'm I'm familiar with like four-bit quantization, quantization, quantization, GG or um GG or um GG or um yeah, BF16, FP8, INT4. yeah, BF16, FP8, INT4. yeah, BF16, FP8, INT4. So is GGML just another flavor of this So is GGML just another flavor of this So is GGML just another flavor of this so to speak? Is it just a different so to speak? Is it just a different so to speak? Is it just a different quantization method? Is it the same as quantization method? Is it the same as quantization method? Is it the same as for example four-bit for example four-bit for example four-bit or or or BF16? Okay, GGML is not a quantization method Okay, GGML is not a quantization method itself. It's a file format. Okay.

Yeah. Yeah. Quantization how you are how you reduce Quantization how you are how you reduce Quantization how you are how you reduce the precision, yeah, exactly. Yes. Yes. These are full precision floating point These are full precision floating point These are full precision floating point formats. No quantization loss. formats. No quantization loss. formats. No quantization loss. A BF16 model is huge, yes, exactly, but A BF16 model is huge, yes, exactly, but A BF16 model is huge, yes, exactly, but it needs VRAM. INT4, yes. So that's it needs VRAM. INT4, yes. So that's it needs VRAM. INT4, yes. So that's smaller and you lose a little bit of smaller and you lose a little bit of smaller and you lose a little bit of performance. performance. performance. Okay. Okay. Yeah. So because it's using GGML Okay. Yeah. So because it's using GGML that is specifically built for Llama that is specifically built for Llama that is specifically built for Llama CPP. Okay, VLM cannot reliably run it CPP. Okay, VLM cannot reliably run it CPP. Okay, VLM cannot reliably run it because GGML support is extremely because GGML support is extremely because GGML support is extremely limited. Okay, so that's why I cannot limited. Okay, so that's why I cannot limited. Okay, so that's why I cannot use GGML. So this note is actually use GGML. So this note is actually use GGML. So this note is actually um It's not a quantization It's not a quantization method.

GGML is a GGML is a what did you call it? A what did you call it? A what did you call it? A storage format. storage format. storage format. It's a file format. It's a file format. It's a file format. Storage container format. Storage container format. Storage container format. Not quantization. So because VLM does not support So because VLM does not support GGML I cannot use that. GGML I cannot use that. GGML I cannot use that. Okay. Even if loaded it would be a bad Even if loaded it would be a bad experience. experience. experience. Okay. What you should do instead? Best option What you should do instead? Best option for you. Mhm.

Yeah, so the issue is that I'm using Yeah, so the issue is that I'm using KServe on Kubernetes and they limit me KServe on Kubernetes and they limit me KServe on Kubernetes and they limit me to VLLM. to VLLM. to VLLM. So I am limited to what I can use with So I am limited to what I can use with So I am limited to what I can use with VLLM. VLLM. VLLM. So can you search on X and all of the So can you search on X and all of the So can you search on X and all of the most recent sources most recent sources most recent sources and documentation for methods how I can and documentation for methods how I can and documentation for methods how I can deploy Qwen 3.6 deploy Qwen 3.6 deploy Qwen 3.6 using VLLM on my 4090 with 24 GB of VRAM. with 24 GB of VRAM. Using VLLM. Look how good the text to speeches of of Look how good the text to speeches of of this. This is amazing. VLM, VRAM this. This is amazing. VLM, VRAM this. This is amazing. VLM, VRAM KServe, it picked all of that up. It's KServe, it picked all of that up. It's KServe, it picked all of that up. It's so good.

Already signed up. Oh, cool, man. Good Already signed up. Oh, cool, man. Good to hear. Try to register for your waiting list, Try to register for your waiting list, did not get an email. Also did not find did not get an email. Also did not find did not get an email. Also did not find it in spam. Really? it in spam. Really? it in spam. Really? Has anyone else signed up and gotten an Has anyone else signed up and gotten an Has anyone else signed up and gotten an email? email? email? Can you please check that? Can you please check that? Can you please check that? Let me actually go in Let me actually go in Let me actually go in um here. Okay, yeah, so I see that eight of you Okay, yeah, so I see that eight of you guys have signed up already. guys have signed up already. guys have signed up already. Let me see if I can Well, I I don't see any Timotei in the Well, I I don't see any Timotei in the subscriber list.

subscriber list. subscriber list. So either you signed up with a different either you signed up with a different name or you are not using your real name or you are not using your real name or you are not using your real name. But name. But name. But um what I do know is that if you enter what I do know is that if you enter like, you know, like, you know, like, you know, these funky alias or something like that these funky alias or something like that these funky alias or something like that it it it usually doesn't work. it it it usually doesn't work. it it it usually doesn't work. So in order to get it you do have to sign in order to get it you do have to sign up with your real up with your real up with your real um um um email. That's the only thing I can email. That's the only thing I can email. That's the only thing I can imagine that is going wrong here cuz imagine that is going wrong here cuz imagine that is going wrong here cuz other people are apparently other people are apparently other people are apparently Let me see. Uh just signed up, got the email. Okay, Uh just signed up, got the email. Okay, yeah. So it is working. That could actually be true if you're That could actually be true if you're already If you're already on the mailing list If you're already on the mailing list that is actually true.

that is actually true. that is actually true. Um Wait, I cannot I should not be sharing Wait, I cannot I should not be sharing those emails on screen. So let let me those emails on screen. So let let me those emails on screen. So let let me just double check that. Sebastian. Just to make sure that things are going Just to make sure that things are going well here. well here. well here. Uh landing page. Yeah, I also don't see a Sebastian in Yeah, I also don't see a Sebastian in that list. So maybe you I did you sign up with your So maybe you I did you sign up with your Did you fill out your first name as Did you fill out your first name as Did you fill out your first name as Oh, hold on. Oh, hold on. Oh, hold on. All subscribers. Yeah, okay. Yeah, okay. Well, now I see a Sebastian K.

Um I won't dox your last name, but Um I won't dox your last name, but Sebastian K, you are now s- Sebastian K, you are now s- Sebastian K, you are now s- showing up as a confirmed subscriber showing up as a confirmed subscriber showing up as a confirmed subscriber here. here. here. But did you not get any mail? But did you not get any mail? But did you not get any mail? Did you see a page Did you see a page Did you see a page where it said it confirmed you as a where it said it confirmed you as a where it said it confirmed you as a member? Um I think you are as a Kubernetes Um I think you are as a Kubernetes engineer, you are uniquely positioned to engineer, you are uniquely positioned to engineer, you are uniquely positioned to make cool AI stuff simply because it make cool AI stuff simply because it make cool AI stuff simply because it seems a Kubernetes cluster is becomes seems a Kubernetes cluster is becomes seems a Kubernetes cluster is becomes essential for AI. Yes, yes, yes. I mean, essential for AI. Yes, yes, yes. I mean, essential for AI. Yes, yes, yes. I mean, that's exactly that's exactly that's exactly uh why uh why uh why like Kubernetes engineers are going to like Kubernetes engineers are going to like Kubernetes engineers are going to make a killing make a killing make a killing um in the next decade or so um in the next decade or so um in the next decade or so because Kubernetes is the de facto operating Kubernetes is the de facto operating system for AI. system for AI. system for AI. Like all of the big Like all of the big Like all of the big AI companies models, it's all running on AI companies models, it's all running on AI companies models, it's all running on Kubernetes. Kubernetes. Kubernetes. Like there's no no doubt about that at Like there's no no doubt about that at Like there's no no doubt about that at all anymore. all anymore. all anymore. Kubernetes has won.

Uh let's see. Uh let's see. There's no case of blessed path for There's no case of blessed path for There's no case of blessed path for onslaught. onslaught. onslaught. Okay. Yeah, okay. So, we what we need to look Yeah, okay. So, we what we need to look out for is an AWQ I got it, but it delayed around 20 I got it, but it delayed around 20 minutes. Okay. Well, minutes. Okay. Well, minutes. Okay. Well, thank you for for thank you for for thank you for for um confirming it though. um confirming it though. um confirming it though. Thank you for that. Now now at least we know that everything Now now at least we know that everything is working properly. is working properly. is working properly. But so Sebastian, just to be sure, like But so Sebastian, just to be sure, like But so Sebastian, just to be sure, like you were already on my mailing list you were already on my mailing list you were already on my mailing list and you signed up for this and you signed up for this and you signed up for this and you got a reconfirmation mail. Could and you got a reconfirmation mail. Could and you got a reconfirmation mail. Could you please confirm that to me? Just to you please confirm that to me? Just to you please confirm that to me? Just to Okay, I'm just learning how all of this Okay, I'm just learning how all of this Okay, I'm just learning how all of this works as I go along, too.

works as I go along, too. works as I go along, too. But so, could you please confirm that? But so, could you please confirm that? But so, could you please confirm that? Um Um Um Peter Kruse, hello. Peter Kruse, hello. Peter Kruse, hello. Uh and running large language models Uh and running large language models Uh and running large language models efficiently. So, probably an optimized efficiently. So, probably an optimized efficiently. So, probably an optimized kind of Postgres for LLMs. Okay, that's kind of Postgres for LLMs. Okay, that's kind of Postgres for LLMs. Okay, that's an interesting way to think about it. I'm not sure if it's actually like I'm not sure if it's actually like database related, but database related, but database related, but Okay. Okay. Okay. Um Um Um So, we need a 27B So, we need a 27B So, we need a 27B AWQ quantization for Qwen 3.6 if this is AWQ quantization for Qwen 3.6 if this is AWQ quantization for Qwen 3.6 if this is going to be going to be going to be feasible at all. So, Qwen 3.6 feasible at all. So, Qwen 3.6 feasible at all. So, Qwen 3.6 27B 27B 27B and then AWQ. Yeah, okay. So, but that is what I tried Yeah, okay. So, but that is what I tried though. But didn't we try But didn't we try already?

Signing up for the waiting list sent no Signing up for the waiting list sent no confirmation e-post. It might take some confirmation e-post. It might take some confirmation e-post. It might take some time, Jarmund. I will um I will um I would see if I can see your name here. Skal sjekke om du Skal sjekke om du står på lista her. Jarmund. Jarmund. Uh Ja. So, Jarmund, du står på lista. Ja. So, Jarmund, du står på lista. Og mailen Og mailen Og mailen ja, blir antageligvis sendt Uh du står som unconfirmed. Uh du står som unconfirmed. Så du må sjekke spam. Så du må sjekke spam. Så du må sjekke spam. Vær så snill. Vær så snill. Vær så snill. Kan hende at det he- det havner der.

Men eller så må vi bare vente litt Men eller så må vi bare vente litt igjen. igjen. igjen. Hvis ikke så kan du bare sende meg en en Hvis ikke så kan du bare sende meg en en Hvis ikke så kan du bare sende meg en en melding på X eller skrive en kommentar melding på X eller skrive en kommentar melding på X eller skrive en kommentar et sted. et sted. et sted. Så ordner seg. I was just saying I was just saying to to to uh uh uh Jarmund that he needs to check his spam. Jarmund that he needs to check his spam. Jarmund that he needs to check his spam. I did check if he is on the list. He is I did check if he is on the list. He is I did check if he is on the list. He is on the list as unconfirmed. on the list as unconfirmed. on the list as unconfirmed. And if it doesn't work out, he can send And if it doesn't work out, he can send And if it doesn't work out, he can send me a message on X or write a comment. me a message on X or write a comment. me a message on X or write a comment. So, it's nice to be able to speak a So, it's nice to be able to speak a So, it's nice to be able to speak a little bit of Norwegian. little bit of Norwegian. little bit of Norwegian. I have to keep it up. I have to keep it up. I have to keep it up. Um let's see. Um let's see. Um let's see. Another confirmation email and it added Another confirmation email and it added Another confirmation email and it added me to the list of founders. Okay, great. me to the list of founders. Okay, great. me to the list of founders. Okay, great. So, now we know exactly that it works. So, now we know exactly that it works. So, now we know exactly that it works. Thank you. Thank you. Thank you. All right. So, SciQ is exactly the model All right. So, SciQ is exactly the model All right. So, SciQ is exactly the model on the unmerged branch commit. on the unmerged branch commit. on the unmerged branch commit. It loaded fine, but only with enforce It loaded fine, but only with enforce It loaded fine, but only with enforce eager eager eager CUDA graph capture arms even at 8K.

So, it was effectively pointing back at So, it was effectively pointing back at the same wall. But there is no Qwen 14B on But there is no Qwen 14B on Qwen Qwen Qwen 3.6 14. That that doesn't exist. 3.6 14. That that doesn't exist. 3.6 14. That that doesn't exist. Yeah. Yeah. Yeah. So, that doesn't exist. So, yeah. For So, that doesn't exist. So, yeah. For So, that doesn't exist. So, yeah. For now, we're just going to stick to Gemma now, we're just going to stick to Gemma now, we're just going to stick to Gemma 4. 4. 4. Please write an Please write an Please write an article article article on why I cannot use on why I cannot use on why I cannot use um Gemma Gemma 27B. 27B. 27B. Go into great detail Go into great detail Go into great detail and and and include the uh technical details uh technical details and and the issues and and the issues and and the issues that we were running into.

Da kan du spamme der. Siste spammen kom Da kan du spamme der. Siste spammen kom 15:04 i går. 15:04 i går. 15:04 i går. Okay. Ja, nei, da kan da må du antageligvis Ja, nei, da kan da må du antageligvis bare vente litt. bare vente litt. bare vente litt. Det kan hende at de sender det ut Det kan hende at de sender det ut Det kan hende at de sender det ut kanskje kanskje to ganger i timen eller kanskje kanskje to ganger i timen eller kanskje kanskje to ganger i timen eller noe sånn. Det kan hende. Okay. So, we now know that 27B is not Okay. So, we now know that 27B is not going to be going to be going to be the the the way forward. It is getting close to my end time here It is getting close to my end time here that I want to um uh I want to get some food and do some uh I want to get some food and do some piano practice piano practice piano practice and sit in the sun. It's actually very and sit in the sun. It's actually very and sit in the sun. It's actually very nice weather out. nice weather out. nice weather out. So, I think this would be a good place So, I think this would be a good place So, I think this would be a good place to to to uh end the work session for right now. uh end the work session for right now. uh end the work session for right now. I might do another session this evening, I might do another session this evening, I might do another session this evening, actually.

Cuz actually I I've been having a lot of Cuz actually I I've been having a lot of fun doing this on stream. It's been fun doing this on stream. It's been fun doing this on stream. It's been really nice to have you guys with me really nice to have you guys with me really nice to have you guys with me here and and asking questions. here and and asking questions. here and and asking questions. I've been definitely I've been definitely I've been definitely um um um considering to do this for a while, not considering to do this for a while, not considering to do this for a while, not knowing for sure if I was going to knowing for sure if I was going to knowing for sure if I was going to stream it or not. stream it or not. stream it or not. And I now And I now And I now yeah, verify that I'm actually having a yeah, verify that I'm actually having a yeah, verify that I'm actually having a good time. good time. good time. And yeah. So, I'll I'll be streaming And yeah. So, I'll I'll be streaming And yeah. So, I'll I'll be streaming when I'm working on this project And I when I'm working on this project And I when I'm working on this project And I think I'm just going to be streaming it. think I'm just going to be streaming it. think I'm just going to be streaming it. So, if you haven't already, So, if you haven't already, So, if you haven't already, please make sure you subscribe. please make sure you subscribe. please make sure you subscribe. And also, And also, And also, um um um if you want to join the community that if you want to join the community that if you want to join the community that I'm maybe launching, I'm maybe launching, I'm maybe launching, then then then check out this link check out this link check out this link um here. um here. um here. I'm I might If there's enough interest, I'm I might If there's enough interest, I'm I might If there's enough interest, I might be launching a new AI community. I might be launching a new AI community. I might be launching a new AI community. So, you can learn AI from me as I'm So, you can learn AI from me as I'm So, you can learn AI from me as I'm building this, too.

building this, too. building this, too. Um But, I'm going to let it write the But, I'm going to let it write the article here. So, we're going to dive article here. So, we're going to dive article here. So, we're going to dive Make sure that we dive deep into why I Make sure that we dive deep into why I Make sure that we dive deep into why I was not able to run 27B on my GPU. And was not able to run 27B on my GPU. And was not able to run 27B on my GPU. And when that is done, and I've taken some when that is done, and I've taken some when that is done, and I've taken some notes, notes, notes, and I will um and I will um and I will um And then, I will probably end the stream And then, I will probably end the stream And then, I will probably end the stream for today. But, I I want to share this because I I But, I I want to share this because I I was I was struggling was I was struggling was I was struggling Well, me and the AI were struggling Well, me and the AI were struggling Well, me and the AI were struggling to get it to work because, like, to get it to work because, like, to get it to work because, like, by all by all by all um um um Like, it really appeared that it should Like, it really appeared that it should Like, it really appeared that it should be able to run this model on be able to run this model on be able to run this model on my 4090, right? my 4090, right? my 4090, right? Let's see. Exit. Okay. Okay. Why we can't run Qwen 27B on CubeCraft Why we can't run Qwen 27B on CubeCraft Why we can't run Qwen 27B on CubeCraft AI.

Short version, the constraint isn't the Short version, the constraint isn't the weight count, it's the quantization or weight count, it's the quantization or weight count, it's the quantization or the KV cache. It's the combination of the KV cache. It's the combination of the KV cache. It's the combination of vLLM and only 24 GB of VRAM. vLLM and only 24 GB of VRAM. vLLM and only 24 GB of VRAM. KServe gives us no choice on choice on KServe gives us no choice on choice on KServe gives us no choice on choice on the first half of that pair. the first half of that pair. the first half of that pair. So, a single node GPU worker So, a single node GPU worker So, a single node GPU worker on my Talos cluster, yep, 4090, after on my Talos cluster, yep, 4090, after on my Talos cluster, yep, 4090, after CUDA, CUDA, CUDA, we have 23 and 1/2 GB allocatable to the we have 23 and 1/2 GB allocatable to the we have 23 and 1/2 GB allocatable to the vLLM process. So, when you load LLMs, vLLM process. So, when you load LLMs, vLLM process. So, when you load LLMs, they are loaded into VRAM, right? they are loaded into VRAM, right? they are loaded into VRAM, right? There's no second GPU. So, I as of yet, There's no second GPU. So, I as of yet, There's no second GPU. So, I as of yet, I I'm working with one GPU at this I I'm working with one GPU at this I I'm working with one GPU at this stage. stage. stage. But, I'm having so much fun that I'm But, I'm having so much fun that I'm But, I'm having so much fun that I'm heavily considering getting a getting a heavily considering getting a getting a heavily considering getting a getting a few more. few more. few more. Um KServe means uh vLLM. So, I'm using Um KServe means uh vLLM. So, I'm using Um KServe means uh vLLM. So, I'm using KServe to to run the AI models on Kubernetes. run the AI models on Kubernetes. run the AI models on Kubernetes. Zero matches for Llama or GGUF. Zero matches for Llama or GGUF. Zero matches for Llama or GGUF. So, basically, So, basically, So, basically, KServe uses vLLM plus LLMD, and that is KServe uses vLLM plus LLMD, and that is KServe uses vLLM plus LLMD, and that is what we're using. So, I'm keeping to what we're using. So, I'm keeping to what we're using. So, I'm keeping to that that that ecosystem architecture choice.

ecosystem architecture choice. ecosystem architecture choice. So, we tried to do the cutover. So, we tried to do the cutover. So, we tried to do the cutover. We first tried 35 billion parameters We first tried 35 billion parameters We first tried 35 billion parameters with the AWQ quantization. That didn't with the AWQ quantization. That didn't with the AWQ quantization. That didn't work. work. work. It didn't even start. Attempt two, 27B. Attempt two, 27B. Technically works, but at nine tokens Technically works, but at nine tokens Technically works, but at nine tokens per second, it was a regression. per second, it was a regression. per second, it was a regression. So, yeah, it did start, but I could only So, yeah, it did start, but I could only So, yeah, it did start, but I could only have a context window of 8,000, which have a context window of 8,000, which have a context window of 8,000, which kind of beats the purpose of running kind of beats the purpose of running kind of beats the purpose of running Qwen 3.6. Qwen 3.6. Qwen 3.6. Throughput collapsed. Throughput collapsed. Throughput collapsed. And And And Mhm. So, vLLM's CUDA graph capture peak. So, So, vLLM's CUDA graph capture peak. So, this is the technical detail that gets this is the technical detail that gets this is the technical detail that gets lost in will it fit. lost in will it fit. lost in will it fit. 4090 has 24 GB to hold all of these 4090 has 24 GB to hold all of these 4090 has 24 GB to hold all of these simultaneously during steady state simultaneously during steady state simultaneously during steady state serving.

serving. serving. Mhm. Yes, it has the Yes, it has the the weights, so the actual model, the KV the weights, so the actual model, the KV the weights, so the actual model, the KV cache, cache, cache, activation buffers, prefill, and the activation buffers, prefill, and the activation buffers, prefill, and the CUDA runtime. CUDA runtime. CUDA runtime. Yeah. Yeah. Yeah. The wall is the CUDA graph capture. The wall is the CUDA graph capture. The wall is the CUDA graph capture. vLLM's default path captures CUDA graphs vLLM's default path captures CUDA graphs vLLM's default path captures CUDA graphs at startup for the most common batch at startup for the most common batch at startup for the most common batch shapes. shapes. shapes. Graph capture is a transient memory Graph capture is a transient memory Graph capture is a transient memory event. The runtime, yeah. event. The runtime, yeah. event. The runtime, yeah. On a 4090 with weights, On a 4090 with weights, On a 4090 with weights, there isn't room for the spike. Okay, I there isn't room for the spike. Okay, I there isn't room for the spike. Okay, I got that. So, Onnx Sloth, yeah, so what's the So, Onnx Sloth, yeah, so what's the difference then between um difference then between um difference then between um Onnx Sloth? Onnx Sloth? Onnx Sloth? Uh Uh Uh Wait, are you from Nordigameisha? Always Wait, are you from Nordigameisha? Always Wait, are you from Nordigameisha? Always thought you Dutch.

thought you Dutch. thought you Dutch. No, I'm I am not I am not from Norway, No, I'm I am not I am not from Norway, No, I'm I am not I am not from Norway, but I did live there for nine years. but I did live there for nine years. but I did live there for nine years. Um When I was 21, I moved there. Um When I was 21, I moved there. Um When I was 21, I moved there. Had a girl there. Had a girl there. Had a girl there. Broke up after a year, and I stayed. Broke up after a year, and I stayed. Broke up after a year, and I stayed. So, I was there for nine years in total. So, I was there for nine years in total. So, I was there for nine years in total. Um Um Um Very much into Very much into Very much into being and living in the mountains. being and living in the mountains. being and living in the mountains. Friluftsliv, as they call it. I was Friluftsliv, as they call it. I was Friluftsliv, as they call it. I was always out in the mountains. always out in the mountains. always out in the mountains. And camping, really enjoyed that life. And camping, really enjoyed that life. And camping, really enjoyed that life. And eventually, I went back. This is And eventually, I went back. This is And eventually, I went back. This is Neovim. Neovim. Neovim. Uh let's see. Why Onnx Sloth GGUF would Uh let's see. Why Onnx Sloth GGUF would Uh let's see. Why Onnx Sloth GGUF would fit and why we still can't use it. So, fit and why we still can't use it. So, fit and why we still can't use it. So, Onnx Sloth Onnx Sloth Onnx Sloth uses GGUF, which is the format that we uses GGUF, which is the format that we uses GGUF, which is the format that we found out. found out. found out. But, tighter packing But, tighter packing But, tighter packing keeps scale packed into the same 4-bit keeps scale packed into the same 4-bit keeps scale packed into the same 4-bit rather than living separately in BF16. rather than living separately in BF16. rather than living separately in BF16. Oh.

So, vLLM So, vLLM loads it separately into memory. loads it separately into memory. loads it separately into memory. And there's no CUDA graph capture peak. And there's no CUDA graph capture peak. And there's no CUDA graph capture peak. Slam CVP use a fundamentally different Slam CVP use a fundamentally different Slam CVP use a fundamentally different memory model. No torch dispatch, no memory model. No torch dispatch, no memory model. No torch dispatch, no compiled graph pre-allocation, compiled graph pre-allocation, compiled graph pre-allocation, and no PyTorch style workspace. and no PyTorch style workspace. and no PyTorch style workspace. You don't pay the transient several You don't pay the transient several You don't pay the transient several gigabyte spike that vLLM does. Same gigabyte spike that vLLM does. Same gigabyte spike that vLLM does. Same model, same card, different runtime, model, same card, different runtime, model, same card, different runtime, different fit. different fit. different fit. Interesting. Interesting. Interesting. I'm learning so much from this. Like, I'm learning so much from this. Like, I'm learning so much from this. Like, I'm loving this so much. Okay. So, what does fit on this hardware Okay. So, what does fit on this hardware via KServe? So, Gemma 4, Qwen 14B, that that doesn't So, Gemma 4, Qwen 14B, that that doesn't exist, though. So, why is it that's exist, though. So, why is it that's exist, though. So, why is it that's hallucinated? hallucinated? hallucinated? Um Okay.

Okay. So, Gemma 26B So, Gemma 26B So, Gemma 26B is working pro That's work working is working pro That's work working is working pro That's work working great. great. great. So, we have to So, we have to So, we have to work We have to wait with work We have to wait with work We have to wait with um um um with Qwen. Yeah, or I need a my with Qwen. Yeah, or I need a my with Qwen. Yeah, or I need a my bigger bigger bigger GPU. GPU. GPU. Okay. Um Why I can't run Why I can't run uh Qwen uh Qwen uh Qwen 3.6 There we go.

There we go. Okay. Documenting this for my own future Documenting this for my own future reference. reference. reference. So, now I know that So, I'll either have to So, I'll either have to figure out figure out figure out or we have to wait until the until there is a Qwen 3.6 weights that until there is a Qwen 3.6 weights that is available for my my particular is available for my my particular is available for my my particular situation using vLLM, situation using vLLM, situation using vLLM, or I will have to, yeah, or I will have to, yeah, or I will have to, yeah, get another GPU. So, so far that is um So, so far that is um Like, I'm okay with that because the Like, I'm okay with that because the Like, I'm okay with that because the application itself is working well with application itself is working well with application itself is working well with Gemma. Gemma. Gemma. I'm just setting up the infrastructure I'm just setting up the infrastructure I'm just setting up the infrastructure so I can switch, right? Uh there we go. Works fine.

Uh there we go. Works fine. Okay. Okay. Okay. So, So, So, until I can switch to a new model, I'm until I can switch to a new model, I'm until I can switch to a new model, I'm going to, yeah, maybe going to, yeah, maybe going to, yeah, maybe I'm going to wait with that and keep I'm going to wait with that and keep I'm going to wait with that and keep keep a lookout on what what's going on keep a lookout on what what's going on keep a lookout on what what's going on on the Quen side. Maybe they come on on the Quen side. Maybe they come on on the Quen side. Maybe they come on with a with a with a uh better quantization that I might be uh better quantization that I might be uh better quantization that I might be able to run. able to run. able to run. Um did you try Helix? No, I did not. All right. Well, guys, All right. Well, guys, 50 people have been online. It's nice. 50 people have been online. It's nice. 50 people have been online. It's nice. It's um It's been really nice for you It's um It's been really nice for you It's um It's been really nice for you guys to tune in. guys to tune in. guys to tune in. I'm happy to hear that guy you guys are I'm happy to hear that guy you guys are I'm happy to hear that guy you guys are interested in hearing about this stuff interested in hearing about this stuff interested in hearing about this stuff and watching me and watching me and watching me um build this. um build this. um build this. So, I might do another work session this So, I might do another work session this So, I might do another work session this evening. evening. evening. And if not, I'll be definitely be And if not, I'll be definitely be And if not, I'll be definitely be streaming again tomorrow. streaming again tomorrow. streaming again tomorrow. So, everybody