Deep dive on LLM Inference at Scale — Harshul Jain, Audible & Tanmay Sah, Independent AI Researcher

Uh so good afternoon everyone. Um my Uh so good afternoon everyone. Um my name is Hershel Jan and he is Tanisha. name is Hershel Jan and he is Tanisha. name is Hershel Jan and he is Tanisha. Uh and we would like to welcome you all Uh and we would like to welcome you all Uh and we would like to welcome you all in this two hours workshop on the LLM in this two hours workshop on the LLM in this two hours workshop on the LLM inference. Uh so the goal of this inference. Uh so the goal of this inference. Uh so the goal of this workshop is to understand this domain workshop is to understand this domain workshop is to understand this domain from the first principles uh dive deeper from the first principles uh dive deeper from the first principles uh dive deeper into it and like understand what's going into it and like understand what's going into it and like understand what's going on throughout the industry. on throughout the industry. on throughout the industry. Uh a bit of background about us. So I am Uh a bit of background about us. So I am Uh a bit of background about us. So I am a senior software engineer at Audible. a senior software engineer at Audible. a senior software engineer at Audible. uh have been building MLA data platforms uh have been building MLA data platforms uh have been building MLA data platforms for the past five years and on the sides for the past five years and on the sides for the past five years and on the sides I have been writing this opensource I have been writing this opensource I have been writing this opensource handbook on LLM inference handbook on LLM inference handbook on LLM inference and Tanme he is the senior quantitative and Tanme he is the senior quantitative and Tanme he is the senior quantitative modeler at XAN cup bank corporation he modeler at XAN cup bank corporation he modeler at XAN cup bank corporation he recently completed his PhD and he has recently completed his PhD and he has recently completed his PhD and he has been actively doing research in the been actively doing research in the been actively doing research in the agent verifiers and the world models Uh so a quick show of hands here. Uh Vu Uh so a quick show of hands here. Uh Vu here is like brand new to the LLM here is like brand new to the LLM here is like brand new to the LLM inference.

inference. inference. Okay, great. And Vu here has like Okay, great. And Vu here has like Okay, great. And Vu here has like deployed these models in production. deployed these models in production. deployed these models in production. They have been tuning it. They have been They have been tuning it. They have been They have been tuning it. They have been serving the production traffic. serving the production traffic. serving the production traffic. Okay, great. Okay, great. Okay, great. So this workshop is targeted towards the So this workshop is targeted towards the So this workshop is targeted towards the beginner and the intermediate level. U beginner and the intermediate level. U beginner and the intermediate level. U and all of the slides and exercises they and all of the slides and exercises they and all of the slides and exercises they are in the repo. I will share that soon. Here is the quick agenda for the Here is the quick agenda for the workshop. We will start with the problem workshop. We will start with the problem workshop. We will start with the problem statement. We will try to understand few statement. We will try to understand few statement. We will try to understand few of the pain points around LLM inference. of the pain points around LLM inference. of the pain points around LLM inference. uh then we understand what causes those uh then we understand what causes those uh then we understand what causes those pain points and build our foundations pain points and build our foundations pain points and build our foundations from there. from there. from there. Then we will dive into like two kind of Then we will dive into like two kind of Then we will dive into like two kind of the optimizations that we do like the the optimizations that we do like the the optimizations that we do like the model optimizations and the serving model optimizations and the serving model optimizations and the serving optimizations. optimizations. optimizations. uh and then we start learning about uh and then we start learning about uh and then we start learning about different serving engines that are different serving engines that are different serving engines that are available to deploy our LLM inference available to deploy our LLM inference available to deploy our LLM inference solutions in production and we will solutions in production and we will solutions in production and we will showcase some benchmarks and the showcase some benchmarks and the showcase some benchmarks and the decision chart on like which engine to decision chart on like which engine to decision chart on like which engine to use.

Cool. So to understand the pain points Cool. So to understand the pain points first we need to know what is like LLM first we need to know what is like LLM first we need to know what is like LLM inference. So, and probably a lot of us inference. So, and probably a lot of us inference. So, and probably a lot of us already know this. Um, but yeah, already know this. Um, but yeah, already know this. Um, but yeah, anything that you ask your AI to do like anything that you ask your AI to do like anything that you ask your AI to do like whether it be generate a video, audio, whether it be generate a video, audio, whether it be generate a video, audio, analyze any text, uh, analyze your analyze any text, uh, analyze your analyze any text, uh, analyze your medical reports or like your tax bills, medical reports or like your tax bills, medical reports or like your tax bills, all of that is like an LLM inference. all of that is like an LLM inference. all of that is like an LLM inference. And this market is like approximately And this market is like approximately And this market is like approximately $23 billion today. $23 billion today. $23 billion today. uh semi analysis recently shared that if uh semi analysis recently shared that if uh semi analysis recently shared that if you want to model like a Google search you want to model like a Google search you want to model like a Google search queries with LLMs, you need like a queries with LLMs, you need like a queries with LLMs, you need like a profit drain of like $36 billion profit drain of like $36 billion profit drain of like $36 billion and query cost has to be less than 0.5 and query cost has to be less than 0.5 and query cost has to be less than 0.5 cents to keep your search business cents to keep your search business cents to keep your search business profitable. profitable. profitable. On the other hand, the business insider On the other hand, the business insider On the other hand, the business insider mentioned like your AI has to be put on mentioned like your AI has to be put on mentioned like your AI has to be put on diet diet diet and everyone has to start auditing and and everyone has to start auditing and and everyone has to start auditing and budgeting their token usage and all of budgeting their token usage and all of budgeting their token usage and all of this is happening. Why? Because your this is happening. Why? Because your this is happening. Why? Because your hardware is limited, compute is hardware is limited, compute is hardware is limited, compute is expensive, your inference is expensive expensive, your inference is expensive expensive, your inference is expensive and with the growing need of like more and with the growing need of like more and with the growing need of like more and more AI usage, this inference cost and more AI usage, this inference cost and more AI usage, this inference cost is rising more and more.

is rising more and more. is rising more and more. So So So this stat, it's an old stat from the this stat, it's an old stat from the this stat, it's an old stat from the open AI, but it's still it's still true. open AI, but it's still it's still true. open AI, but it's still it's still true. So if you look at the like training cost So if you look at the like training cost So if you look at the like training cost of the GPT3, it was like around $4.6 of the GPT3, it was like around $4.6 of the GPT3, it was like around $4.6 million. It was a one-time cost. But if million. It was a one-time cost. But if million. It was a one-time cost. But if you see the inference cost that has been you see the inference cost that has been you see the inference cost that has been like uh it's a recurring cost because like uh it's a recurring cost because like uh it's a recurring cost because it's a operating cost that scales with it's a operating cost that scales with it's a operating cost that scales with every user that comes in that every every user that comes in that every every user that comes in that every token that comes in every session that token that comes in every session that token that comes in every session that uh is being initiated on the like AI and there are only two ways to basically and there are only two ways to basically counter this. Uh one way is you reduce counter this. Uh one way is you reduce counter this. Uh one way is you reduce your token usage. your token usage. your token usage. um alternative is you should try to um alternative is you should try to um alternative is you should try to optimize your inference solutions as a optimize your inference solutions as a optimize your inference solutions as a inference service provider for your inference service provider for your inference service provider for your customers and for yourself. And so this customers and for yourself. And so this customers and for yourself. And so this um we have been seeing like lot and lot um we have been seeing like lot and lot um we have been seeing like lot and lot of like new solutions coming out every of like new solutions coming out every of like new solutions coming out every then and now. Um and so the idea would then and now. Um and so the idea would then and now. Um and so the idea would be like okay we will try to build those be like okay we will try to build those be like okay we will try to build those foundations that will help us understand foundations that will help us understand foundations that will help us understand and evaluate like whatever ships next.

and evaluate like whatever ships next. and evaluate like whatever ships next. Um so yeah to get started like we will Um so yeah to get started like we will Um so yeah to get started like we will do a quick demo like it's a short demo do a quick demo like it's a short demo do a quick demo like it's a short demo of like what are the different pain of like what are the different pain of like what are the different pain points around inference points around inference points around inference u and so this is the repo uh I mean you u and so this is the repo uh I mean you u and so this is the repo uh I mean you can pull it or you can also open it on can pull it or you can also open it on can pull it or you can also open it on the GitHub the GitHub the GitHub uh it's called LLM inference at scale a uh it's called LLM inference at scale a uh it's called LLM inference at scale a bit of background here like four months bit of background here like four months bit of background here like four months back when I didn't knew anything on the back when I didn't knew anything on the back when I didn't knew anything on the LLM inference um I started learning it I LLM inference um I started learning it I LLM inference um I started learning it I saw like lot of resources were saw like lot of resources were saw like lot of resources were scattered. So we started putting it uh scattered. So we started putting it uh scattered. So we started putting it uh like all together in one place uh so like all together in one place uh so like all together in one place uh so that it could benefit people. that it could benefit people. that it could benefit people. Um yes. So let me actually get out of Um yes. So let me actually get out of Um yes. So let me actually get out of this slideshow mode and probably this slideshow mode and probably this slideshow mode and probably go into I will go to this extended mode. I will go to this extended mode. Um okay great.

okay great. Uh yeah so in this uh repository if you Uh yeah so in this uh repository if you Uh yeah so in this uh repository if you see a readme file there is like a link see a readme file there is like a link see a readme file there is like a link to the slides. Uh so this it will be to the slides. Uh so this it will be to the slides. Uh so this it will be like this folder where you have like a like this folder where you have like a like this folder where you have like a pptx and there is like a benchmark pptx and there is like a benchmark pptx and there is like a benchmark report in there. report in there. report in there. uh you can always like download it and uh you can always like download it and uh you can always like download it and then for the demo purposes uh we have then for the demo purposes uh we have then for the demo purposes uh we have couple of Jupyter notebooks. Uh we have couple of Jupyter notebooks. Uh we have couple of Jupyter notebooks. Uh we have like collaborated with Moab who are the like collaborated with Moab who are the like collaborated with Moab who are the like Google collab alternative like Google collab alternative like Google collab alternative and what they basically provide you is and what they basically provide you is and what they basically provide you is like a free RTX 6000 GPU. It's a 100 GB like a free RTX 6000 GPU. It's a 100 GB like a free RTX 6000 GPU. It's a 100 GB V RAM GPU. V RAM GPU. V RAM GPU. So So So and we have like already set up these and we have like already set up these and we have like already set up these notebooks so that it becomes easy to notebooks so that it becomes easy to notebooks so that it becomes easy to like experiment with and like all of the like experiment with and like all of the like experiment with and like all of the assets and everything are preset for assets and everything are preset for assets and everything are preset for you. Uh you. Uh you. Uh so uh we will start with like a simple so uh we will start with like a simple so uh we will start with like a simple demo a demo a demo a probably Okay.

Okay. Um yeah. So Um yeah. So Um yeah. So when it comes to the inference, you need when it comes to the inference, you need when it comes to the inference, you need to do an inference on a certain model, to do an inference on a certain model, to do an inference on a certain model, right? Uh for the workshop purposes, we right? Uh for the workshop purposes, we right? Uh for the workshop purposes, we are using a simple ML 7B model. Uh it's are using a simple ML 7B model. Uh it's are using a simple ML 7B model. Uh it's a small model of around 15GB in size. So a small model of around 15GB in size. So a small model of around 15GB in size. So we we are going to like load that into we we are going to like load that into we we are going to like load that into the GPU. So, the GPU. So, the GPU. So, and we would look like some of the GPU and we would look like some of the GPU and we would look like some of the GPU stats as well. So, we see like okay, we stats as well. So, we see like okay, we stats as well. So, we see like okay, we are working on the 6,000 Blackwell. Uh, are working on the 6,000 Blackwell. Uh, are working on the 6,000 Blackwell. Uh, and you might be thinking I'm not and you might be thinking I'm not and you might be thinking I'm not running the cells because I don't trust running the cells because I don't trust running the cells because I don't trust the Wi-Fi at conferences. the Wi-Fi at conferences. the Wi-Fi at conferences. So, yeah. So, I would probably be just So, yeah. So, I would probably be just So, yeah. So, I would probably be just going over the results uh that we kind going over the results uh that we kind going over the results uh that we kind of ran previously. Uh, yeah. So, we have like a GPU which Uh, yeah. So, we have like a GPU which is like 102GB. is like 102GB. is like 102GB. Uh now the first thing that comes to my Uh now the first thing that comes to my Uh now the first thing that comes to my mind is like what's my memory mind is like what's my memory mind is like what's my memory consumption looks like when I do the LLM consumption looks like when I do the LLM consumption looks like when I do the LLM inference. So I load this model inference. So I load this model inference. So I load this model uh and I see like okay I have like a 15 uh and I see like okay I have like a 15 uh and I see like okay I have like a 15 GB here. So I have roughly like 87.5GB.

GB here. So I have roughly like 87.5GB. GB here. So I have roughly like 87.5GB. And now when I do the like inference And now when I do the like inference And now when I do the like inference here here here uh what I notice is like the more the uh what I notice is like the more the uh what I notice is like the more the number of inputs I pass more is the number of inputs I pass more is the number of inputs I pass more is the memory that I need. memory that I need. memory that I need. uh and it's increasing slowly but it's uh and it's increasing slowly but it's uh and it's increasing slowly but it's still increasing. [snorts] So imagine still increasing. [snorts] So imagine still increasing. [snorts] So imagine like if you have a context length of like if you have a context length of like if you have a context length of like around 4,000 or 16,000 or 32,000 like around 4,000 or 16,000 or 32,000 like around 4,000 or 16,000 or 32,000 uh tokens. Uh so this memory could like uh tokens. Uh so this memory could like uh tokens. Uh so this memory could like really grow big and it you could really grow big and it you could really grow big and it you could actually get like all of those out of actually get like all of those out of actually get like all of those out of memory issues. memory issues. memory issues. Uh so definitely this is like your Uh so definitely this is like your Uh so definitely this is like your problem one like your memory increasing problem one like your memory increasing problem one like your memory increasing with the increase in tokens. So in form with the increase in tokens. So in form with the increase in tokens. So in form of like a simple visualization it looks of like a simple visualization it looks of like a simple visualization it looks like this. The second problem that you would see is The second problem that you would see is like like like the time to your first token it's very the time to your first token it's very the time to your first token it's very very slow. Uh we measure it by a metric very slow. Uh we measure it by a metric very slow. Uh we measure it by a metric called TTFT. It's a short short form of called TTFT. It's a short short form of called TTFT. It's a short short form of it. uh and when you try to like measure it. uh and when you try to like measure it. uh and when you try to like measure the TTFT uh with the like input size you the TTFT uh with the like input size you the TTFT uh with the like input size you would see like longer the context would see like longer the context would see like longer the context you would see like this uh TTFT being you would see like this uh TTFT being you would see like this uh TTFT being slow. So now there are two problems.

slow. So now there are two problems. slow. So now there are two problems. Your memory increases with the token Your memory increases with the token Your memory increases with the token size. Your TTFT increases with the token size. Your TTFT increases with the token size. Your TTFT increases with the token size. size. size. Uh sorry not the token size, the context Uh sorry not the token size, the context Uh sorry not the token size, the context size. Uh and then the third is the like Uh and then the third is the like throughput. The throughput is like how throughput. The throughput is like how throughput. The throughput is like how many tokens can you serve per second and many tokens can you serve per second and many tokens can you serve per second and then how many users can you serve per then how many users can you serve per then how many users can you serve per second. So if you take a very very second. So if you take a very very second. So if you take a very very vanilla implementation on your local vanilla implementation on your local vanilla implementation on your local system system system uh it would be like very sequential. So uh it would be like very sequential. So uh it would be like very sequential. So if you send like five requests all those if you send like five requests all those if you send like five requests all those five requests would be catered like five requests would be catered like five requests would be catered like sequentially rather than parallelly. sequentially rather than parallelly. sequentially rather than parallelly. Uh and so like your request basically Uh and so like your request basically Uh and so like your request basically takes more time to complete if you have takes more time to complete if you have takes more time to complete if you have like multiple users. like multiple users. like multiple users. So these are the like three problems. So these are the like three problems. So these are the like three problems. There is a fourth one. I haven't There is a fourth one. I haven't There is a fourth one. I haven't described it here. probably we will described it here. probably we will described it here. probably we will build that intuition as we move forward. build that intuition as we move forward. build that intuition as we move forward. Uh but let's remember like these are the Uh but let's remember like these are the Uh but let's remember like these are the three problems the memory TTFT and the three problems the memory TTFT and the three problems the memory TTFT and the throughput.

throughput. throughput. Cool. Uh I will go back to the slides. Oh, you you need the Oh, you you need the Oh, okay. Workshop. Okay. Yeah. So, within that repository, if you Yeah. So, within that repository, if you see a workshop folder, you see that see a workshop folder, you see that see a workshop folder, you see that readme and then the readme has all the readme and then the readme has all the readme and then the readme has all the links, the slides and the demos.

Okay. uh so let's start working through Okay. uh so let's start working through the foundations like let's start the foundations like let's start the foundations like let's start understanding understanding understanding uh what are the reasons behind those uh what are the reasons behind those uh what are the reasons behind those pain points and for that like we have to pain points and for that like we have to pain points and for that like we have to look at this inference pipeline look at this inference pipeline look at this inference pipeline um um um so we get like an input text uh that so we get like an input text uh that so we get like an input text uh that text could have like any number of words text could have like any number of words text could have like any number of words you convert those into the tokens so for you convert those into the tokens so for you convert those into the tokens so for simplicity you can assume one word equal simplicity you can assume one word equal simplicity you can assume one word equal to one token to one token to one token uh then you kind of convert them into uh then you kind of convert them into uh then you kind of convert them into like the embeddings and then you send it like the embeddings and then you send it like the embeddings and then you send it to the like transformers to the like transformers to the like transformers uh like there are 32 layers of uh like there are 32 layers of uh like there are 32 layers of transformers but that's specific to the transformers but that's specific to the transformers but that's specific to the Mistl 7B different models have different Mistl 7B different models have different Mistl 7B different models have different kind number of layers uh and then you kind number of layers uh and then you kind number of layers uh and then you generate a new token and that token generate a new token and that token generate a new token and that token basically goes back to the input then basically goes back to the input then basically goes back to the input then you generate another token and that you generate another token and that you generate another token and that keeps on going now in this entire keeps on going now in this entire keeps on going now in this entire pipeline you would see like 95% of your pipeline you would see like 95% of your pipeline you would see like 95% of your compute is like taken by these compute is like taken by these compute is like taken by these transformer layers. So it's worth transformer layers. So it's worth transformer layers. So it's worth looking at like what goes within this looking at like what goes within this looking at like what goes within this transformer layer.

transformer layer. transformer layer. Within this transformer layer you would Within this transformer layer you would Within this transformer layer you would have like more layers. You have like a have like more layers. You have like a have like more layers. You have like a normalization layer, you have an normalization layer, you have an normalization layer, you have an attention layer, you have a feed forward attention layer, you have a feed forward attention layer, you have a feed forward layer and all. layer and all. layer and all. And attention layer is the one I think And attention layer is the one I think And attention layer is the one I think that has been very very famous. that has been very very famous. that has been very very famous. Attention is all you need paper. I think Attention is all you need paper. I think Attention is all you need paper. I think that's very well known. So attention is that's very well known. So attention is that's very well known. So attention is the most compute intensive layer and we the most compute intensive layer and we the most compute intensive layer and we need to understand what goes within that need to understand what goes within that need to understand what goes within that attention layer. attention layer. attention layer. So what does attention do? Attention uh So what does attention do? Attention uh So what does attention do? Attention uh so if you have an input text it needs to so if you have an input text it needs to so if you have an input text it needs to find the attention scores of every token find the attention scores of every token find the attention scores of every token with respect to all of the previous with respect to all of the previous with respect to all of the previous tokens. tokens. tokens. And to do that what it needs to do is And to do that what it needs to do is And to do that what it needs to do is like it needs to project every token like it needs to project every token like it needs to project every token into like a key query and the value into like a key query and the value into like a key query and the value space. So in like in a simpler terms space. So in like in a simpler terms space. So in like in a simpler terms just u understand this like if you have just u understand this like if you have just u understand this like if you have 10 tokens then it needs like the 10 10 tokens then it needs like the 10 10 tokens then it needs like the 10 different query key and the value different query key and the value different query key and the value vectors. If there are 100 tokens you vectors. If there are 100 tokens you vectors. If there are 100 tokens you would need 100 key and the value would need 100 key and the value would need 100 key and the value vectors. If there are thousand tokens vectors. If there are thousand tokens vectors. If there are thousand tokens you would need thousand key value you would need thousand key value you would need thousand key value vectors. And so like your number of the vectors. And so like your number of the vectors. And so like your number of the key and the value vectors they increase key and the value vectors they increase key and the value vectors they increase as you increase the input size.

Uh and if you calculate the like KV size Uh and if you calculate the like KV size per token per token per token uh for a mist 7B it comes out to be 131 uh for a mist 7B it comes out to be 131 uh for a mist 7B it comes out to be 131 KV. Uh this is because like KV. Uh this is because like KV. Uh this is because like uh you have two vectors K and V. You uh you have two vectors K and V. You uh you have two vectors K and V. You have to multiply the size uh one vector have to multiply the size uh one vector have to multiply the size uh one vector is like 128 dimensions. you have to is like 128 dimensions. you have to is like 128 dimensions. you have to multi multiply it by 32 transformer multi multiply it by 32 transformer multi multiply it by 32 transformer layers and then you have to multiply it layers and then you have to multiply it layers and then you have to multiply it by the KV heads for ML 7B it's gate KV by the KV heads for ML 7B it's gate KV by the KV heads for ML 7B it's gate KV heads it's not like 32 because it uses a heads it's not like 32 because it uses a heads it's not like 32 because it uses a different kind of an attention mechanism different kind of an attention mechanism different kind of an attention mechanism u which we will talk about for sure but u which we will talk about for sure but u which we will talk about for sure but yeah so the KV size per token is like yeah so the KV size per token is like yeah so the KV size per token is like your 131 KB now imagine if you have 4K your 131 KB now imagine if you have 4K your 131 KB now imagine if you have 4K context uh So that size becomes like context uh So that size becomes like context uh So that size becomes like half a GB. Uh if you do like 16k context half a GB. Uh if you do like 16k context half a GB. Uh if you do like 16k context that size becomes 2.1 GB. Uh now that size becomes 2.1 GB. Uh now that size becomes 2.1 GB. Uh now multiplay by the users like assume you multiplay by the users like assume you multiplay by the users like assume you can serve multiple users together can serve multiple users together can serve multiple users together at the same time within that GPU you at the same time within that GPU you at the same time within that GPU you could have like 42GB with a 4K context could have like 42GB with a 4K context could have like 42GB with a 4K context and 80 users. Um and if your GPU is only and 80 users. Um and if your GPU is only and 80 users. Um and if your GPU is only like let's say 24GB, you are already like let's say 24GB, you are already like let's say 24GB, you are already running out of the memory. So you cannot running out of the memory. So you cannot running out of the memory. So you cannot serve that many users with that many serve that many users with that many serve that many users with that many context.

To visualize this, look at a GPU memory. To visualize this, look at a GPU memory. So the GPU memory has like a model So the GPU memory has like a model So the GPU memory has like a model weights which are pretty fixed. These weights which are pretty fixed. These weights which are pretty fixed. These are pre-trained weights. Uh there is are pre-trained weights. Uh there is are pre-trained weights. Uh there is like an overhead that is also fixed. like an overhead that is also fixed. like an overhead that is also fixed. that also like that changes but it does that also like that changes but it does that also like that changes but it does not change that much u overall you can not change that much u overall you can not change that much u overall you can assume it's fixed and then there is like assume it's fixed and then there is like assume it's fixed and then there is like a leftover memory so this leftover a leftover memory so this leftover a leftover memory so this leftover memory is what being used by your KB me memory is what being used by your KB me memory is what being used by your KB me like key and the value vectors so like key and the value vectors so like key and the value vectors so assume like you have a one user you can assume like you have a one user you can assume like you have a one user you can only serve that many key and the value only serve that many key and the value only serve that many key and the value vectors or that many tokens which can vectors or that many tokens which can vectors or that many tokens which can like fit in this entire 80GB uh like like fit in this entire 80GB uh like like fit in this entire 80GB uh like memory that is left. Uh Uh so we can show this with a simple demo so we can show this with a simple demo so we can show this with a simple demo too. Uh Okay, great.

Okay, great. Uh, let me see if I can actually run Uh, let me see if I can actually run Uh, let me see if I can actually run this. this. this. What probably? probably? Where the heck is this happening? Okay, great. Okay, great. Yeah. So you would see like the GPU is Yeah. So you would see like the GPU is Yeah. So you would see like the GPU is attached. So here we are just trying to confirm So here we are just trying to confirm the like memory based on the maths and the like memory based on the maths and the like memory based on the maths and based on the intuition that we have based on the intuition that we have based on the intuition that we have built. So the model memory is like let's built. So the model memory is like let's built. So the model memory is like let's say if you have a 7 billion parameters say if you have a 7 billion parameters say if you have a 7 billion parameters you are doing a 16 bit precision your you are doing a 16 bit precision your you are doing a 16 bit precision your total memory comes out to be 14.6 GB total memory comes out to be 14.6 GB total memory comes out to be 14.6 GB with you can basically verify that with with you can basically verify that with with you can basically verify that with the maths. So if you do all that math the maths. So if you do all that math the maths. So if you do all that math that comes out to be the 14.6GB.

that comes out to be the 14.6GB. that comes out to be the 14.6GB. Uh now comes the KVK and the KV size. So Uh now comes the KVK and the KV size. So Uh now comes the KVK and the KV size. So this KV size is like your 131 KB per this KV size is like your 131 KB per this KV size is like your 131 KB per token. token. token. Uh and if you do that maths and you try Uh and if you do that maths and you try Uh and if you do that maths and you try to like visualize this. to like visualize this. to like visualize this. [sighs] Okay. And then let's just visualize Okay. And then let's just visualize this. this. this. Okay. Great. Okay. Great. Okay. Great. Uh yeah so this is the like a memory Uh yeah so this is the like a memory Uh yeah so this is the like a memory chart. So if you see like as your chart. So if you see like as your chart. So if you see like as your context increases your memory keeps context increases your memory keeps context increases your memory keeps increasing. increasing. increasing. Then another thing to realize is like as Then another thing to realize is like as Then another thing to realize is like as your users increase your users increase your users increase uh then also your memory increases. So uh then also your memory increases. So uh then also your memory increases. So if you want to serve like 160 users if you want to serve like 160 users if you want to serve like 160 users uh on a GPU you can support like uh you uh on a GPU you can support like uh you uh on a GPU you can support like uh you can only support like a lesser context can only support like a lesser context can only support like a lesser context length. So there is always a tradeoff length. So there is always a tradeoff length. So there is always a tradeoff between what context length you can between what context length you can between what context length you can serve versus how much cost you can save serve versus how much cost you can save serve versus how much cost you can save by like putting your multiple by like putting your multiple by like putting your multiple uh users or the concurrent users into uh users or the concurrent users into uh users or the concurrent users into like a single GPU. So you have to always like a single GPU. So you have to always like a single GPU. So you have to always take that tradeoff and and we will go take that tradeoff and and we will go take that tradeoff and and we will go through that like uh in couple of more through that like uh in couple of more through that like uh in couple of more slides.

Cool. Cool. Uh okay. Great. So let me pull back. Uh okay. Great. So let me pull back. Uh okay. Great. So let me pull back. So that was like memory. Uh we need to So that was like memory. Uh we need to So that was like memory. Uh we need to understand uh why we had like uh slower understand uh why we had like uh slower understand uh why we had like uh slower time to first token time to first token time to first token uh when we increase the context length. uh when we increase the context length. uh when we increase the context length. So for that like we need to understand So for that like we need to understand So for that like we need to understand the two phases of inference and those the two phases of inference and those the two phases of inference and those phases are like the prefill and the phases are like the prefill and the phases are like the prefill and the decode phase. I think you would all seen decode phase. I think you would all seen decode phase. I think you would all seen like a lot of articles but we just like a lot of articles but we just like a lot of articles but we just wanted to explain it. Uh so when you wanted to explain it. Uh so when you wanted to explain it. Uh so when you send like a lot of uh like when you send send like a lot of uh like when you send send like a lot of uh like when you send these input tokens these input tokens these input tokens what you want to do is uh you want to what you want to do is uh you want to what you want to do is uh you want to build those key and the value vectors build those key and the value vectors build those key and the value vectors that I mentioned for all the tokens. that I mentioned for all the tokens. that I mentioned for all the tokens. Then you want to compute the attention Then you want to compute the attention Then you want to compute the attention scores of every token with respect to scores of every token with respect to scores of every token with respect to the previous token. All this operation the previous token. All this operation the previous token. All this operation that you do it's a very very metricsh that you do it's a very very metricsh that you do it's a very very metricsh heavy uh it's a very very compute heavy heavy uh it's a very very compute heavy heavy uh it's a very very compute heavy operation and we all know like the GPUs operation and we all know like the GPUs operation and we all know like the GPUs they are like very well suited for a they are like very well suited for a they are like very well suited for a heavy compute workload so we call like a heavy compute workload so we call like a heavy compute workload so we call like a prefill to be like a compute bound uh prefill to be like a compute bound uh prefill to be like a compute bound uh and it does take some time to complete and it does take some time to complete and it does take some time to complete so whatever time that this phase takes so whatever time that this phase takes so whatever time that this phase takes to complete that's your time to the to complete that's your time to the to complete that's your time to the first token first token first token So if if you have like a more input So if if you have like a more input So if if you have like a more input tokens, you have to generate more key tokens, you have to generate more key tokens, you have to generate more key value vectors. You have to do lot more value vectors. You have to do lot more value vectors. You have to do lot more attention math and because of that your

attention math and because of that your attention math and because of that your TTFT becomes more more slower. TTFT becomes more more slower. TTFT becomes more more slower. Where is if once you generate one token Where is if once you generate one token Where is if once you generate one token you need to keep doing this to generate you need to keep doing this to generate you need to keep doing this to generate another tokens sequentially one after another tokens sequentially one after another tokens sequentially one after another. But in that process every time another. But in that process every time another. But in that process every time you have to build the key and the value you have to build the key and the value you have to build the key and the value vectors of all the previous tokens vectors of all the previous tokens vectors of all the previous tokens which is same as prefill like you were which is same as prefill like you were which is same as prefill like you were building key value vectors there also building key value vectors there also building key value vectors there also here also but in decode phase you are here also but in decode phase you are here also but in decode phase you are only computing the attention math for only computing the attention math for only computing the attention math for the new token the new token the new token and that is why it's a very less it's and that is why it's a very less it's and that is why it's a very less it's lesser compute oriented lesser compute oriented lesser compute oriented and it's also called as memory bound. We and it's also called as memory bound. We and it's also called as memory bound. We will see it shortly why it's called as will see it shortly why it's called as will see it shortly why it's called as memory bound. So in a classic timeline you would see So in a classic timeline you would see prefill and decode phase like this. So prefill and decode phase like this. So prefill and decode phase like this. So time taken by prefill that's your time time taken by prefill that's your time time taken by prefill that's your time to first token and then your time taken to first token and then your time taken to first token and then your time taken by every decode step that's your by every decode step that's your by every decode step that's your uh basically your inter token latency.

uh basically your inter token latency. uh basically your inter token latency. So that's like the fourth metric So that's like the fourth metric So that's like the fourth metric uh that you need to worry about like uh that you need to worry about like uh that you need to worry about like what's the time being taken by your what's the time being taken by your what's the time being taken by your decode step. Okay, cool. Uh Okay, cool. Uh now why why does the like decode step or now why why does the like decode step or now why why does the like decode step or why does decode takes time and why it's why does decode takes time and why it's why does decode takes time and why it's being called as like a memory bound being called as like a memory bound being called as like a memory bound operation? Let's try to understand that. operation? Let's try to understand that. operation? Let's try to understand that. uh to understand that we need to look at uh to understand that we need to look at uh to understand that we need to look at how the metrics map basically works on how the metrics map basically works on how the metrics map basically works on the GPU on a high level. So GPU has two the GPU on a high level. So GPU has two the GPU on a high level. So GPU has two kind of memories. You have a high kind of memories. You have a high kind of memories. You have a high bandwidth memory. You have a shared bandwidth memory. You have a shared bandwidth memory. You have a shared memory. memory. memory. So the high bandwidth memory is a larger So the high bandwidth memory is a larger So the high bandwidth memory is a larger size but a lower me like lower size but a lower me like lower size but a lower me like lower bandwidth. bandwidth. bandwidth. By lower bandwidth I mean like you can By lower bandwidth I mean like you can By lower bandwidth I mean like you can transfer data out of it at a lower rate transfer data out of it at a lower rate transfer data out of it at a lower rate compared to the shared memory. So the compared to the shared memory. So the compared to the shared memory. So the shared memory is smaller in size but it shared memory is smaller in size but it shared memory is smaller in size but it has a very very high bandwidth. Uh that has a very very high bandwidth. Uh that has a very very high bandwidth. Uh that means you can transfer data in and out means you can transfer data in and out means you can transfer data in and out of it with a very first thing. So when a of it with a very first thing. So when a of it with a very first thing. So when a when you have to do a metric math so you when you have to do a metric math so you when you have to do a metric math so you have to pick the data in chunks from the have to pick the data in chunks from the have to pick the data in chunks from the high bandwidth memory you have to put it high bandwidth memory you have to put it high bandwidth memory you have to put it into the shared memory.

into the shared memory. into the shared memory. Do that math write back the result into Do that math write back the result into Do that math write back the result into the high bandwidth memory. uh for the prefill phase when you have uh for the prefill phase when you have to do this you have to do this matrix to do this you have to do this matrix to do this you have to do this matrix math only once but for the decode phase math only once but for the decode phase math only once but for the decode phase you have to do this metric math uh like you have to do this metric math uh like you have to do this metric math uh like uh again and again because you're uh again and again because you're uh again and again because you're generating each and every token generating each and every token generating each and every token sequentially sequentially sequentially and so like you it doesn't matter like and so like you it doesn't matter like and so like you it doesn't matter like how fast is your decode how fast is your decode how fast is your decode because now you can transfer your data because now you can transfer your data because now you can transfer your data out of the high bandwidth memory into out of the high bandwidth memory into out of the high bandwidth memory into the S shared memory at a certain speed the S shared memory at a certain speed the S shared memory at a certain speed because you are limited by the high because you are limited by the high because you are limited by the high bandwidth memory bandwidth speed bandwidth memory bandwidth speed bandwidth memory bandwidth speed and so that governs your like token and so that governs your like token and so that governs your like token sealing like at what rate can you sealing like at what rate can you sealing like at what rate can you actually generate tokens out of the actually generate tokens out of the actually generate tokens out of the decode step.

If you look at this in the roof line If you look at this in the roof line plot plot plot uh so there is a left section which is uh so there is a left section which is uh so there is a left section which is called to be a memory bound. called to be a memory bound. called to be a memory bound. Mathematically it's governed by the Mathematically it's governed by the Mathematically it's governed by the arithmetic intensity. arithmetic intensity. arithmetic intensity. Arithmetic intensity is the number of Arithmetic intensity is the number of Arithmetic intensity is the number of flip-flop operations that you perform flip-flop operations that you perform flip-flop operations that you perform per bite of data being transferred. So per bite of data being transferred. So per bite of data being transferred. So for the decode step for the decode step for the decode step uh decode step since you are uh decode step since you are uh decode step since you are transferring lot of data like the key transferring lot of data like the key transferring lot of data like the key and the value vectors of the all the and the value vectors of the all the and the value vectors of the all the previous tokens the model weights but previous tokens the model weights but previous tokens the model weights but you are doing the like less computation you are doing the like less computation you are doing the like less computation because you're computing attention math because you're computing attention math because you're computing attention math for only one token. Uh so it's for only one token. Uh so it's for only one token. Uh so it's arithmetic intensity is very low but for arithmetic intensity is very low but for arithmetic intensity is very low but for a prefill phase you are transferring the a prefill phase you are transferring the a prefill phase you are transferring the data once but then like you are doing data once but then like you are doing data once but then like you are doing this heavy computation and so it's this heavy computation and so it's this heavy computation and so it's arithmetic intensity is very high. So arithmetic intensity is very high. So arithmetic intensity is very high. So now you know like in terms of now you know like in terms of now you know like in terms of mathematics like why the computer like mathematics like why the computer like mathematics like why the computer like why the arithmetic intensity of prefill why the arithmetic intensity of prefill why the arithmetic intensity of prefill is very high compared to your decor.

Uh [snorts] Uh [snorts] okay okay okay so this is like another small small so this is like another small small so this is like another small small demo. Uh demo. Uh demo. Uh every time I have to Okay. Okay, great. Okay, great. I hope this is already running. So yeah, I hope this is already running. So yeah, I hope this is already running. So yeah, again we are loading the model. again we are loading the model. again we are loading the model. Now this is the like the prefill cost. Now this is the like the prefill cost. Now this is the like the prefill cost. So what we are basically doing is uh we So what we are basically doing is uh we So what we are basically doing is uh we are getting the like um the input uh are getting the like um the input uh are getting the like um the input uh text and then we are trying to generate text and then we are trying to generate text and then we are trying to generate this um the prefill step the amount of this um the prefill step the amount of this um the prefill step the amount of time it takes. We see like as we time it takes. We see like as we time it takes. We see like as we increase the like size of the input increase the like size of the input increase the like size of the input tokens this prefill is increasing. So tokens this prefill is increasing. So tokens this prefill is increasing. So you and this is the reason why your TDF you and this is the reason why your TDF you and this is the reason why your TDF increases increases increases and then like your decode time. So the and then like your decode time. So the and then like your decode time. So the decode time is like on average it stays decode time is like on average it stays decode time is like on average it stays about the same. Uh and so if it is like about the same. Uh and so if it is like about the same. Uh and so if it is like assuming like you ignore the like cold assuming like you ignore the like cold assuming like you ignore the like cold start your decode time is like start your decode time is like start your decode time is like approximately around the average line.

approximately around the average line. approximately around the average line. it it it it is still impacted by like u it it it it is still impacted by like u it it it it is still impacted by like u the input size. It's not like it's a the input size. It's not like it's a the input size. It's not like it's a constant uh time and it is because it constant uh time and it is because it constant uh time and it is because it still needs to pull the key and the still needs to pull the key and the still needs to pull the key and the value vectors from the memory for all value vectors from the memory for all value vectors from the memory for all the previous tokens. So there is still the previous tokens. So there is still the previous tokens. So there is still like uh that uh basically small increase like uh that uh basically small increase like uh that uh basically small increase in time that you would see with the in time that you would see with the in time that you would see with the decode step. And then this is the like decode step. And then this is the like decode step. And then this is the like classic uh roof line plot. classic uh roof line plot. classic uh roof line plot. Uh okay. Okay. Five. Okay. Great. Okay. Five. Okay. Great. Okay. So Okay. So Okay. So now now let's try to understand like uh now now let's try to understand like uh now now let's try to understand like uh the throughput dimension. You want to the throughput dimension. You want to the throughput dimension. You want to understand how many users you can understand how many users you can understand how many users you can actually serve and I think we saw like a actually serve and I think we saw like a actually serve and I think we saw like a diagram of the GPU memory where we saw diagram of the GPU memory where we saw diagram of the GPU memory where we saw okay there is some memory that is free okay there is some memory that is free okay there is some memory that is free for the key and the value vectors to for the key and the value vectors to for the key and the value vectors to grow.

grow. grow. So assume like you have just a single So assume like you have just a single So assume like you have just a single user user user uh uh uh what's the total KV size that you have what's the total KV size that you have what's the total KV size that you have you can basically support it's defined you can basically support it's defined you can basically support it's defined by your context limit. by your context limit. by your context limit. uh the max users that you can support is uh the max users that you can support is uh the max users that you can support is like whatever is your GPU uh like whatever is your GPU uh like whatever is your GPU uh availability like whatever is the memory availability like whatever is the memory availability like whatever is the memory that is available in the GPU you divide that is available in the GPU you divide that is available in the GPU you divide it by the key and the value size per it by the key and the value size per it by the key and the value size per user uh and when you do that like it user uh and when you do that like it user uh and when you do that like it comes out to be like your the concurrent comes out to be like your the concurrent comes out to be like your the concurrent users. users. users. Now assume like your GPU is fixed, your Now assume like your GPU is fixed, your Now assume like your GPU is fixed, your model is fixed. model is fixed. model is fixed. Uh Uh Uh so so so your KV size per token is fixed. There your KV size per token is fixed. There your KV size per token is fixed. There are only two dimensions that are left are only two dimensions that are left are only two dimensions that are left here which is context and your here which is context and your here which is context and your concurrent users. concurrent users. concurrent users. If you want to serve more concurrent If you want to serve more concurrent If you want to serve more concurrent users, you have to reduce the context users, you have to reduce the context users, you have to reduce the context length. If you reduce the context length. If you reduce the context length. If you reduce the context length, you could impact your quality.

length, you could impact your quality. length, you could impact your quality. Uh so these are the two dimensions right Uh so these are the two dimensions right Uh so these are the two dimensions right now that we are trading off. now that we are trading off. now that we are trading off. Then if we but can you actually serve Then if we but can you actually serve Then if we but can you actually serve the like max number of concurrent users? the like max number of concurrent users? the like max number of concurrent users? Uh in an ideal world probably not Uh in an ideal world probably not Uh in an ideal world probably not because because because every business has like a latency SLO every business has like a latency SLO every business has like a latency SLO that we have to meet. that we have to meet. that we have to meet. So So So if you remember like in the decode step if you remember like in the decode step if you remember like in the decode step I said the time for the decode still I said the time for the decode still I said the time for the decode still increases if you have more inputs. increases if you have more inputs. increases if you have more inputs. It also increases if you have more It also increases if you have more It also increases if you have more users. users. users. So ultimately So ultimately So ultimately uh your inter token latency also gets uh your inter token latency also gets uh your inter token latency also gets impacted impacted impacted if you have like a higher batch size and if you have like a higher batch size and if you have like a higher batch size and your TTF also gets impacted. So now your TTF also gets impacted. So now your TTF also gets impacted. So now there is a third dimension you have to there is a third dimension you have to there is a third dimension you have to worry about which is like your latency. worry about which is like your latency. worry about which is like your latency. So the three dimensions that you have is So the three dimensions that you have is So the three dimensions that you have is like a quality latency and the like a quality latency and the like a quality latency and the throughput. So it comes out to be like throughput. So it comes out to be like throughput. So it comes out to be like this trade-off triangle where you have this trade-off triangle where you have this trade-off triangle where you have to choose between the two. So for a to choose between the two. So for a to choose between the two. So for a premium chat application premium chat application premium chat application you would want to prioritize definitely you would want to prioritize definitely you would want to prioritize definitely the quality and you want to prioritize the quality and you want to prioritize the quality and you want to prioritize the like the latency. You would not want the like the latency. You would not want the like the latency. You would not want your users to wait infinitely for the your users to wait infinitely for the your users to wait infinitely for the like or like not infinitely but probably like or like not infinitely but probably like or like not infinitely but probably for the larger latency.

for the larger latency. for the larger latency. You can always sacrifice the number of You can always sacrifice the number of You can always sacrifice the number of users you can support on the GPU and users you can support on the GPU and users you can support on the GPU and probably take that costed being more probably take that costed being more probably take that costed being more customers customers customers in form of like and and like if you in form of like and and like if you in form of like and and like if you consider like an agent uh sorry the consider like an agent uh sorry the consider like an agent uh sorry the async agent workload you would want to async agent workload you would want to async agent workload you would want to like prioritize definitely quality and like prioritize definitely quality and like prioritize definitely quality and the throughput the throughput the throughput uh because these are the longunning uh because these are the longunning uh because these are the longunning tasks tasks tasks uh and you would want to like serve as uh and you would want to like serve as uh and you would want to like serve as many as concurrent tasks. fast as many as concurrent tasks. fast as many as concurrent tasks. fast as possible but with a very very higher possible but with a very very higher possible but with a very very higher quality. And often like we think like okay if the And often like we think like okay if the GPU is like a very expensive GPU GPU is like a very expensive GPU GPU is like a very expensive GPU uh that might not be a good fit for us. uh that might not be a good fit for us. uh that might not be a good fit for us. Uh but it turns out that could actually Uh but it turns out that could actually Uh but it turns out that could actually serve you the lowest cost per million uh serve you the lowest cost per million uh serve you the lowest cost per million uh tokens. tokens. tokens. Uh but you really have to trust your Uh but you really have to trust your Uh but you really have to trust your kind of calculations on the max users kind of calculations on the max users kind of calculations on the max users that you want and like uh you really that you want and like uh you really that you want and like uh you really have to make those estimations uh have to make those estimations uh have to make those estimations uh correctly.

so we do have like uh so we do have like uh let me just Where is this? Where is this? Okay, great. So, for the capacity Okay, great. So, for the capacity Okay, great. So, for the capacity calculator, uh there is like a link to calculator, uh there is like a link to calculator, uh there is like a link to the collab because I was facing certain the collab because I was facing certain the collab because I was facing certain issues with molab. I had to migrate out issues with molab. I had to migrate out issues with molab. I had to migrate out the wall widget library and I didn't the wall widget library and I didn't the wall widget library and I didn't have time. So, being lazy, I just picked have time. So, being lazy, I just picked have time. So, being lazy, I just picked collab there. Uh apologies to Moab. Uh Uh so my VR is connected. So what we have done over here is we So what we have done over here is we have like shaded some like the GPUs with have like shaded some like the GPUs with have like shaded some like the GPUs with their V RAMs, bandwidths, the flip-flops their V RAMs, bandwidths, the flip-flops their V RAMs, bandwidths, the flip-flops and the cost per hours. Um and the cost per hours. Um and the cost per hours. Um then we kind of like built this simple then we kind of like built this simple then we kind of like built this simple uh like uh capacity calculator. This is uh like uh capacity calculator. This is uh like uh capacity calculator. This is just a KV visualizer uh where you kind just a KV visualizer uh where you kind just a KV visualizer uh where you kind of like when you increase the number of

of like when you increase the number of of like when you increase the number of tokens uh you see like your KV size it tokens uh you see like your KV size it tokens uh you see like your KV size it increases and when you increase the increases and when you increase the increases and when you increase the number of users your size is like number of users your size is like number of users your size is like increasing at a much faster rate increasing at a much faster rate increasing at a much faster rate and then and then and then in this capacity calculator uh let it run. let it run. So we have like a model which we which So we have like a model which we which So we have like a model which we which is like a 7 billion parameter model that is like a 7 billion parameter model that is like a 7 billion parameter model that we selected. we selected. we selected. We set the like precision to be FP16. Uh We set the like precision to be FP16. Uh We set the like precision to be FP16. Uh now we decide the way we basically go by now we decide the way we basically go by now we decide the way we basically go by the GPU decision is you have to decide the GPU decision is you have to decide the GPU decision is you have to decide what's your like you have to fix one what's your like you have to fix one what's your like you have to fix one dimension first which you care about the dimension first which you care about the dimension first which you care about the most. most. most. for premium chat I mentioned like for premium chat I mentioned like for premium chat I mentioned like latency is definitely the one latency is definitely the one latency is definitely the one uh and then like for the async workloads uh and then like for the async workloads uh and then like for the async workloads the batch the minimum batch size that the batch the minimum batch size that the batch the minimum batch size that you want to serve for from like a single you want to serve for from like a single you want to serve for from like a single GPU that is the second dimension so you GPU that is the second dimension so you GPU that is the second dimension so you want to fix these first so I will go want to fix these first so I will go want to fix these first so I will go about like in a premium chat application about like in a premium chat application about like in a premium chat application uh uh uh so I can go ahead with like 10 so I can go ahead with like 10 so I can go ahead with like 10 milliseconds latency a minimum batch milliseconds latency a minimum batch milliseconds latency a minimum batch size I don't care like I can so I'm okay size I don't care like I can so I'm okay size I don't care like I can so I'm okay with like probably two with like probably two with like probably two uh uh uh okay so probably with the seven

okay so probably with the seven okay so probably with the seven concurrent users on a single GPU and concurrent users on a single GPU and concurrent users on a single GPU and then like my context limit is very then like my context limit is very then like my context limit is very important to me because I want to focus important to me because I want to focus important to me because I want to focus on the quality as well on the quality as well on the quality as well uh and so like I do see like some of the uh and so like I do see like some of the uh and so like I do see like some of the GPUs so the H18GB GPUs so the H18GB GPUs so the H18GB it's like a $8 per hour but like am I it's like a $8 per hour but like am I it's like a $8 per hour but like am I 300x is it? Yeah. So it's like around 300x is it? Yeah. So it's like around 300x is it? Yeah. So it's like around $10 per hour but if you do all that $10 per hour but if you do all that $10 per hour but if you do all that throughput math that we shared in the throughput math that we shared in the throughput math that we shared in the mathematics before you could find like mathematics before you could find like mathematics before you could find like your cost per million dollar tokens that your cost per million dollar tokens that your cost per million dollar tokens that could be very very that could be like could be very very that could be like could be very very that could be like lesser. So lesser. So lesser. So you need to do such calculations by you need to do such calculations by you need to do such calculations by fixing those dimensions and you need to fixing those dimensions and you need to fixing those dimensions and you need to decide your GPU to like reduce your kind decide your GPU to like reduce your kind decide your GPU to like reduce your kind of inference cost. This is at least the of inference cost. This is at least the of inference cost. This is at least the first step that you can take towards first step that you can take towards first step that you can take towards optimizing the inference.

So the next slide. So let me So the next slide. So let me Okay, great. Okay, great. Okay, great. And so like now the next thing is about And so like now the next thing is about And so like now the next thing is about the model optimization. So we are now the model optimization. So we are now the model optimization. So we are now basically have built that foundation basically have built that foundation basically have built that foundation where we understood some of the pain where we understood some of the pain where we understood some of the pain points, reason behind those pain points, points, reason behind those pain points, points, reason behind those pain points, why those were happening why those were happening why those were happening um how we could like address that GPU um how we could like address that GPU um how we could like address that GPU capacity thing. We need to understand capacity thing. We need to understand capacity thing. We need to understand what can we do like what can we further what can we do like what can we further what can we do like what can we further do about it. So it it is about the model do about it. So it it is about the model do about it. So it it is about the model optimization and I think I would like to optimization and I think I would like to optimization and I think I would like to invite Tan I he can talk more about invite Tan I he can talk more about invite Tan I he can talk more about these model optimizations provided he these model optimizations provided he these model optimizations provided he has worked uh on this like during his has worked uh on this like during his has worked uh on this like during his research times research times research times okay I can control yeah here okay hi everyone uh mic check yeah here okay hi everyone uh mic check am I audible at last yeah okay so hi I'm am I audible at last yeah okay so hi I'm am I audible at last yeah okay so hi I'm Tesha I work as a senior quant modeler Tesha I work as a senior quant modeler Tesha I work as a senior quant modeler and also I am an AI researcher. My work and also I am an AI researcher. My work and also I am an AI researcher. My work focuses on a agent verification and focuses on a agent verification and focuses on a agent verification and right now building world models. So for right now building world models. So for right now building world models. So for this one model optimization this one model optimization this one model optimization before we start model optimization so I before we start model optimization so I before we start model optimization so I created a research template so that it created a research template so that it created a research template so that it will be easy for us to understand all will be easy for us to understand all will be easy for us to understand all these complex things. I so our template these complex things. I so our template these complex things. I so our template is simple. First we will identify the is simple. First we will identify the is simple. First we will identify the problem. Second step we will solve the problem. Second step we will solve the problem. Second step we will solve the problem using two algorithms. These are problem using two algorithms. These are problem using two algorithms. These are just fake algorithms. So first algorithm just fake algorithms. So first algorithm just fake algorithms. So first algorithm is called ostrich algorithm. Whenever we

is called ostrich algorithm. Whenever we is called ostrich algorithm. Whenever we see uh just like ostrich whenever we see see uh just like ostrich whenever we see see uh just like ostrich whenever we see a problem ostrich put their head into a problem ostrich put their head into a problem ostrich put their head into the sand. So same thing we will do the sand. So same thing we will do the sand. So same thing we will do whenever we face a problem we will just whenever we face a problem we will just whenever we face a problem we will just ignore it. So this is an important ignore it. So this is an important ignore it. So this is an important algorithm we should follow. Second one algorithm we should follow. Second one algorithm we should follow. Second one is created it is called world cup is created it is called world cup is created it is called world cup algorithm. For example, we don't know algorithm. For example, we don't know algorithm. For example, we don't know who will win this FIFA World Cup. So, who will win this FIFA World Cup. So, who will win this FIFA World Cup. So, what organizers did, they uh break the what organizers did, they uh break the what organizers did, they uh break the 48 teams into 12 groups, uh then round 48 teams into 12 groups, uh then round 48 teams into 12 groups, uh then round 32. So, round 32 right now is currently 32. So, round 32 right now is currently 32. So, round 32 right now is currently going on. Uh then round 16, then going on. Uh then round 16, then going on. Uh then round 16, then quarterfinals, uh then semi-finals and quarterfinals, uh then semi-finals and quarterfinals, uh then semi-finals and finals. So what they are doing is that finals. So what they are doing is that finals. So what they are doing is that uh they are breaking it into a smaller uh they are breaking it into a smaller uh they are breaking it into a smaller problems and the useful results are problems and the useful results are problems and the useful results are moving forward. So same analogy or same moving forward. So same analogy or same moving forward. So same analogy or same algorithm we will use uh to understand algorithm we will use uh to understand algorithm we will use uh to understand this model optimization all those this model optimization all those this model optimization all those things. So yeah let's start. So I have things. So yeah let's start. So I have things. So yeah let's start. So I have one H100 GPU.

one H100 GPU. one H100 GPU. I have to use this open-source model I have to use this open-source model I have to use this open-source model what is called GPTOSS what is called GPTOSS what is called GPTOSS 120 billion parameter model. So right 120 billion parameter model. So right 120 billion parameter model. So right now I think it's so they have trained it now I think it's so they have trained it now I think it's so they have trained it on BF float 16 and weight is 240 GB. on BF float 16 and weight is 240 GB. on BF float 16 and weight is 240 GB. What should I do? What should I do? What should I do? This is the problem we have. So first This is the problem we have. So first This is the problem we have. So first thing what we have to deal do is that thing what we have to deal do is that thing what we have to deal do is that 240 GB and 80 uh GB H100. 240 GB and 80 uh GB H100. 240 GB and 80 uh GB H100. So and I have to fit only in one GPU or So and I have to fit only in one GPU or So and I have to fit only in one GPU or not in multiple GPU. So what can we do? not in multiple GPU. So what can we do? not in multiple GPU. So what can we do? I think simple step is that just I think simple step is that just I think simple step is that just compress it. But how should we compress compress it. But how should we compress compress it. But how should we compress it? Uh that's the another challenge. So it? Uh that's the another challenge. So it? Uh that's the another challenge. So if we compress BF BF float 16 to FP8 uh if we compress BF BF float 16 to FP8 uh if we compress BF BF float 16 to FP8 uh then it will be around 120 GB but our then it will be around 120 GB but our then it will be around 120 GB but our GPU H100 is still 80 GB. So what I think GPU H100 is still 80 GB. So what I think GPU H100 is still 80 GB. So what I think they did is that they compressed it into they did is that they compressed it into they did is that they compressed it into further MX uh MX FP4 and I think size is further MX uh MX FP4 and I think size is further MX uh MX FP4 and I think size is around 65 GB. So this is something we around 65 GB. So this is something we around 65 GB. So this is something we can do uh compress but question so and can do uh compress but question so and can do uh compress but question so and we will use over this ostrich algorithm we will use over this ostrich algorithm we will use over this ostrich algorithm we are assuming that uh there is no loss we are assuming that uh there is no loss we are assuming that uh there is no loss in compressing a bigger model into a a in compressing a bigger model into a a in compressing a bigger model into a a smaller size. Second thing smaller size. Second thing smaller size. Second thing in uh in this one okay yeah so in this

in uh in this one okay yeah so in this in uh in this one okay yeah so in this one in this slide we have used this one in this slide we have used this one in this slide we have used this mistral 7B so 7 billion parameters so mistral 7B so 7 billion parameters so mistral 7B so 7 billion parameters so it's a small model 7 billion parameters it's a small model 7 billion parameters it's a small model 7 billion parameters so uh so if you multiply it by two bytes so uh so if you multiply it by two bytes so uh so if you multiply it by two bytes so it so weight of it's around is 14 so it so weight of it's around is 14 so it so weight of it's around is 14 14.5 GB which can easily fit into H100 14.5 GB which can easily fit into H100 14.5 GB which can easily fit into H100 or even a a40 so or even a a40 so or even a a40 so so Next uh what we can do is that like so Next uh what we can do is that like so Next uh what we can do is that like mistral 7B instead of compressing it a mistral 7B instead of compressing it a mistral 7B instead of compressing it a floating point 16 we can apply different floating point 16 we can apply different floating point 16 we can apply different techniques like int 8 or int4 or nf4. So techniques like int 8 or int4 or nf4. So techniques like int 8 or int4 or nf4. So basically we have to just use ostrich basically we have to just use ostrich basically we have to just use ostrich algorithm and just believe that uh there algorithm and just believe that uh there algorithm and just believe that uh there is no quality loss kind of things but is no quality loss kind of things but is no quality loss kind of things but somehow we also have to mathematically somehow we also have to mathematically somehow we also have to mathematically prove that by doing some kind of test prove that by doing some kind of test prove that by doing some kind of test testing on some external benchmark that testing on some external benchmark that testing on some external benchmark that whether it is working or not. So the and whether it is working or not. So the and whether it is working or not. So the and this comes under post training this comes under post training this comes under post training quantization kind of thing. One can also quantization kind of thing. One can also quantization kind of thing. One can also do uh this one uh during finetuning one do uh this one uh during finetuning one do uh this one uh during finetuning one can also do this kind of quantization.

can also do this kind of quantization. can also do this kind of quantization. This comes under a quant training kind This comes under a quant training kind This comes under a quant training kind of thing. So uh let's move to our next of thing. So uh let's move to our next of thing. So uh let's move to our next problem. problem. problem. So we have this huge matrices just just we have this huge matrices just just imagine imagine uh 1,000 by 1,000 uh imagine imagine uh 1,000 by 1,000 uh imagine imagine uh 1,000 by 1,000 uh dimension matrix A and another matrix dimension matrix A and another matrix dimension matrix A and another matrix matrix um 1,000 by 1,000. So if we matrix um 1,000 by 1,000. So if we matrix um 1,000 by 1,000. So if we multiply uh if we multiply by this two multiply uh if we multiply by this two multiply uh if we multiply by this two matrices so number of operations will be matrices so number of operations will be matrices so number of operations will be 1,000 raised to the power q and this is 1,000 raised to the power q and this is 1,000 raised to the power q and this is kind of a problem in terms of uh uh in kind of a problem in terms of uh uh in kind of a problem in terms of uh uh in terms of computing. So we wondered our terms of computing. So we wondered our terms of computing. So we wondered our matrix multiplication should be fast and matrix multiplication should be fast and matrix multiplication should be fast and it should save memory. So what should we it should save memory. So what should we it should save memory. So what should we do? We have a giant matrix. Okay, let's do? We have a giant matrix. Okay, let's do? We have a giant matrix. Okay, let's take this one. Uh, Mr. 4096 by 4096.

take this one. Uh, Mr. 4096 by 4096. take this one. Uh, Mr. 4096 by 4096. What should we do uh to What should we do uh to What should we do uh to solve our problem of speeding up the solve our problem of speeding up the solve our problem of speeding up the things and saving the memory 4096 by things and saving the memory 4096 by things and saving the memory 4096 by 4096. 4096. 4096. So first thing is that we will use just So first thing is that we will use just So first thing is that we will use just our world cup algorithm. We can decide a our world cup algorithm. We can decide a our world cup algorithm. We can decide a random number just break the block random number just break the block random number just break the block vertically. It does not matter what you vertically. It does not matter what you vertically. It does not matter what you are choosing it. So you have so let's are choosing it. So you have so let's are choosing it. So you have so let's say uh we have 4096 uh columns we will say uh we have 4096 uh columns we will say uh we have 4096 uh columns we will break it uh we will break it into a break it uh we will break it into a break it uh we will break it into a group of 128 column each. So 128 128 128 group of 128 column each. So 128 128 128 group of 128 column each. So 128 128 128 128 uh vertical vert uh vertically so we 128 uh vertical vert uh vertically so we 128 uh vertical vert uh vertically so we will get a 30 we will get this 32 blocks will get a 30 we will get this 32 blocks will get a 30 we will get this 32 blocks if we divide this 4096 if we divide this 4096 if we divide this 4096 then then then what will happen by doing this thing? So what will happen by doing this thing? So what will happen by doing this thing? So if we just divide this one vertical if we just divide this one vertical if we just divide this one vertical vertically then we can use a multiple vertically then we can use a multiple vertically then we can use a multiple GPU to speed up the process. So this GPU to speed up the process. So this GPU to speed up the process. So this kind of thing is called multi head kind of thing is called multi head kind of thing is called multi head attention.

attention. attention. So what else can we do? We have a big So what else can we do? We have a big So what else can we do? We have a big matrix like matrix like matrix like uh as I have mentioned that ostrich uh as I have mentioned that ostrich uh as I have mentioned that ostrich algorithm. So our main problem is algorithm. So our main problem is algorithm. So our main problem is sizing. So what we what we can do is sizing. So what we what we can do is sizing. So what we what we can do is that instead of having all those 32 uh that instead of having all those 32 uh that instead of having all those 32 uh 32 vertical blocks we will throw away uh 32 vertical blocks we will throw away uh 32 vertical blocks we will throw away uh 31 blocks and we will assume that one 31 blocks and we will assume that one 31 blocks and we will assume that one block is sufficient enough that all the block is sufficient enough that all the block is sufficient enough that all the queries uh can handle those blocks. Our queries uh can handle those blocks. Our queries uh can handle those blocks. Our loss will be almost negligible and we loss will be almost negligible and we loss will be almost negligible and we come up with this algorithm uh which uh come up with this algorithm uh which uh come up with this algorithm uh which uh and this algorithm is called a and this algorithm is called a and this algorithm is called a multiquery attention. So as we can see multiquery attention. So as we can see multiquery attention. So as we can see right now we are at two spectrum. One is right now we are at two spectrum. One is right now we are at two spectrum. One is multi head attention where we split it multi head attention where we split it multi head attention where we split it into 32 blocks and use different uh into 32 blocks and use different uh into 32 blocks and use different uh different uh GPUs or do some parallel different uh GPUs or do some parallel different uh GPUs or do some parallel processing and at the same time we are processing and at the same time we are processing and at the same time we are just throwing 31 blocks and uh we are just throwing 31 blocks and uh we are just throwing 31 blocks and uh we are calling this is as a multi-query calling this is as a multi-query calling this is as a multi-query attention. So attention. So attention. So uh so at both extreme we should be come uh so at both extreme we should be come uh so at both extreme we should be come up with a middle ground like something up with a middle ground like something up with a middle ground like something we can say that we can say that we can say that instead of throwing all the 31 uh maybe instead of throwing all the 31 uh maybe instead of throwing all the 31 uh maybe we can group we can group we can group we can group we can group we can group we can group we can group we can group some of the blocks together so that uh some of the blocks together so that uh some of the blocks together so that uh uh and we can assume that uh similar uh and we can assume that uh similar uh and we can assume that uh similar blocks will attend to a um similar kind blocks will attend to a um similar kind blocks will attend to a um similar kind of uh queries. So this kind of technique

of uh queries. So this kind of technique of uh queries. So this kind of technique comes under grouped query attention comes under grouped query attention comes under grouped query attention which is very popular right now. Uh even which is very popular right now. Uh even which is very popular right now. Uh even in uh even in mistral or in other models in uh even in mistral or in other models in uh even in mistral or in other models this grouped query attention works. So this grouped query attention works. So this grouped query attention works. So right now we have understand that we right now we have understand that we right now we have understand that we have a big matrix uh we can divide it have a big matrix uh we can divide it have a big matrix uh we can divide it the way we want and doing some the way we want and doing some the way we want and doing some mathematical calculation prove that loss mathematical calculation prove that loss mathematical calculation prove that loss is almost negligible kind of thing. So is almost negligible kind of thing. So is almost negligible kind of thing. So what else we can do? what else we can do? what else we can do? So after that uh after this grouped So after that uh after this grouped So after that uh after this grouped query attention query attention query attention uh uh uh see see see uh we have a big matrix uh one is one is uh we have a big matrix uh one is one is uh we have a big matrix uh one is one is key and one is value. key and one is value. key and one is value. Let's compress that matrix into a latent Let's compress that matrix into a latent Let's compress that matrix into a latent vector and then come up with some vector and then come up with some vector and then come up with some algorithm to uh reconstruct from latent algorithm to uh reconstruct from latent algorithm to uh reconstruct from latent vector uh to our original matrix. So vector uh to our original matrix. So vector uh to our original matrix. So this kind of a strategy comes under this this kind of a strategy comes under this this kind of a strategy comes under this one um multi head latent latent one um multi head latent latent one um multi head latent latent attention but again it has some problems attention but again it has some problems attention but again it has some problems with rope because rope is position with rope because rope is position with rope because rope is position dependent and uh and it is position dependent and uh and it is position dependent and uh and it is position independent kind of thing. So yeah one independent kind of thing. So yeah one independent kind of thing. So yeah one needs to also include some uh index for needs to also include some uh index for needs to also include some uh index for keys also so that one can map it. But keys also so that one can map it. But keys also so that one can map it. But again main problem is that why why we again main problem is that why why we again main problem is that why why we are why we are multiplying all those big are why we are multiplying all those big are why we are multiplying all those big matrices. So because that's how this matrices. So because that's how this matrices. So because that's how this attention mechanism works that

attention mechanism works that attention mechanism works that each token will pay attention to every each token will pay attention to every each token will pay attention to every token. So how about let's don't pay token. So how about let's don't pay token. So how about let's don't pay attention to all the previous token only attention to all the previous token only attention to all the previous token only pay attention to the important tokens uh pay attention to the important tokens uh pay attention to the important tokens uh which is important for us. So this is a which is important for us. So this is a which is important for us. So this is a kind of uh this kind of field is uh kind of uh this kind of field is uh kind of uh this kind of field is uh evolving. So this comes under sparse uh evolving. So this comes under sparse uh evolving. So this comes under sparse uh deepseek sparse attention. So deepseek sparse attention. So deepseek sparse attention. So uh yeah and yeah yeah yeah so okay next uh yeah and yeah yeah yeah so okay next uh yeah and yeah yeah yeah so okay next yeah so next one is flash attention. So yeah so next one is flash attention. So yeah so next one is flash attention. So uh so in flash attention so main pro so uh so in flash attention so main pro so uh so in flash attention so main pro so main problem is that uh main problem is that uh main problem is that uh uh so so currently so so currently not uh so so currently so so currently not uh so so currently so so currently not currently so right now almost everyone currently so right now almost everyone currently so right now almost everyone uses flash attention but way in 2022 or uses flash attention but way in 2022 or uses flash attention but way in 2022 or 2023 uh so that's how it works that's 2023 uh so that's how it works that's 2023 uh so that's how it works that's how it works is that uh so how it works is that uh so how it works is that uh so uh this uh this uh this Q K query and A and key matrices they Q K query and A and key matrices they Q K query and A and key matrices they were in HBM. Uh it loads uh it uh first were in HBM. Uh it loads uh it uh first were in HBM. Uh it loads uh it uh first uh it loads into uh this one uh tensor uh it loads into uh this one uh tensor uh it loads into uh this one uh tensor core and it do some uh it do some core and it do some uh it do some core and it do some uh it do some calculation and then it will uh write it calculation and then it will uh write it calculation and then it will uh write it back to uh HBM and then uh this process back to uh HBM and then uh this process back to uh HBM and then uh this process goes on multiple times. So in flash goes on multiple times. So in flash goes on multiple times. So in flash attention uh what they did is that attention uh what they did is that attention uh what they did is that uh is that instead of multiplying the uh is that instead of multiplying the uh is that instead of multiplying the whole matrices so they just divided it

whole matrices so they just divided it whole matrices so they just divided it into like our world cup algorithm into like our world cup algorithm into like our world cup algorithm divided the bigger matrices into a small divided the bigger matrices into a small divided the bigger matrices into a small tile and only put those small tiles uh tile and only put those small tiles uh tile and only put those small tiles uh into a SBM so that uh it can process into a SBM so that uh it can process into a SBM so that uh it can process multiplication fast and just uh keep uh multiplication fast and just uh keep uh multiplication fast and just uh keep uh keeping track of this some three keeping track of this some three keeping track of this some three variables so that they can calculate variables so that they can calculate variables so that they can calculate this online softmax. Yeah. Yeah. Next one. So, yeah. So, so this is just Next one. So, yeah. So, so this is just Next one. So, yeah. So, so this is just mathematics. So, if we have a multi head mathematics. So, if we have a multi head mathematics. So, if we have a multi head attention if it is 524 attention if it is 524 attention if it is 524 uh KV uh then it depends upon how much uh KV uh then it depends upon how much uh KV uh then it depends upon how much how much grouping we want and so if how much grouping we want and so if how much grouping we want and so if instead of 32 KV head we only want to instead of 32 KV head we only want to instead of 32 KV head we only want to use uh 8 KV heads. So uh so so we can use uh 8 KV heads. So uh so so we can use uh 8 KV heads. So uh so so we can get a compression of 4x times and this get a compression of 4x times and this get a compression of 4x times and this multi head latent attention this formula multi head latent attention this formula multi head latent attention this formula depends on the model to model how many depends on the model to model how many depends on the model to model how many layers your model have. So in the layers your model have. So in the layers your model have. So in the original deepseek paper uh I think they original deepseek paper uh I think they original deepseek paper uh I think they have some 128 dimension have some 128 dimension have some 128 dimension 128 d 12 I don't remember the exact 128 d 12 I don't remember the exact 128 d 12 I don't remember the exact dimension but according to that uh they dimension but according to that uh they dimension but according to that uh they have used uh this one latent vector in have used uh this one latent vector in have used uh this one latent vector in which they have used 512 as a dimension which they have used 512 as a dimension which they have used 512 as a dimension and some 64 and some 64 and some 64 for for rope index. So and then they for for rope index. So and then they for for rope index. So and then they show that it is a 50x 56x show that it is a 50x 56x show that it is a 50x 56x uh more compressed than multi head

uh more compressed than multi head uh more compressed than multi head attention. Okay. Okay. Yeah. So uh so yeah so this is uh so Yeah. So uh so yeah so this is uh so Yeah. So uh so yeah so this is uh so this is the uh this is the trade-off uh this is the uh this is the trade-off uh this is the uh this is the trade-off uh trade-off diagram. So here I think we trade-off diagram. So here I think we trade-off diagram. So here I think we have not talked about this linear have not talked about this linear have not talked about this linear attention or mamba. So main problem is attention or mamba. So main problem is attention or mamba. So main problem is just all this m Matrix multiplication. just all this m Matrix multiplication. just all this m Matrix multiplication. Right now everyone is using attention. Right now everyone is using attention. Right now everyone is using attention. Suppose in future Suppose in future Suppose in future uh if we don't want to use attention or uh if we don't want to use attention or uh if we don't want to use attention or rather than generating tokens rather than generating tokens rather than generating tokens sequentially just use maybe diffusion sequentially just use maybe diffusion sequentially just use maybe diffusion models where we can generate everything models where we can generate everything models where we can generate everything simultaneously. So all these algorithms simultaneously. So all these algorithms simultaneously. So all these algorithms will change also. But here I think they will change also. But here I think they will change also. But here I think they have two more. One is linear attention have two more. One is linear attention have two more. One is linear attention and one is mamba. So according to uh and one is mamba. So according to uh and one is mamba. So according to uh this slide so this slide so this slide so if we are not compressing anything so if we are not compressing anything so if we are not compressing anything so MHA is just we are parallelizing the MHA is just we are parallelizing the MHA is just we are parallelizing the process so there is no quality loss so process so there is no quality loss so process so there is no quality loss so it's a good and then this uh grouped it's a good and then this uh grouped it's a good and then this uh grouped query attention which is I think almost query attention which is I think almost query attention which is I think almost uh every model is using uh just GQA and uh every model is using uh just GQA and uh every model is using uh just GQA and DSA kind of thing or yeah DSA kind of thing or yeah DSA kind of thing or yeah I think same thing we are providing in I think same thing we are providing in I think same thing we are providing in the attention mechanism scorecard So uh the attention mechanism scorecard So uh the attention mechanism scorecard So uh so I think uh this one mha quality is so I think uh this one mha quality is so I think uh this one mha quality is good throughput is uh throughput is okay good throughput is uh throughput is okay good throughput is uh throughput is okay and for grouped query attention it and for grouped query attention it and for grouped query attention it depends upon your use case also though

depends upon your use case also though depends upon your use case also though yeah though yeah though yeah though quality is almost similar to uh multi quality is almost similar to uh multi quality is almost similar to uh multi head attention but use case also matters head attention but use case also matters head attention but use case also matters a lot yeah multi-query attention is just a lot yeah multi-query attention is just a lot yeah multi-query attention is just one extreme we are one extreme we are one extreme we are I don't know why but we are just I don't know why but we are just I don't know why but we are just assuming that we only need one block and assuming that we only need one block and assuming that we only need one block and all the queries will attend to that all the queries will attend to that all the queries will attend to that smaller smaller block. So, so quality is smaller smaller block. So, so quality is smaller smaller block. So, so quality is not that great for M for MQA and this not that great for M for MQA and this not that great for M for MQA and this multi head latent attention. So yeah if multi head latent attention. So yeah if multi head latent attention. So yeah if you have tried some this deep seat you have tried some this deep seat you have tried some this deep seat models so I think uh they are doing models so I think uh they are doing models so I think uh they are doing great job yeah in in quality wise great job yeah in in quality wise great job yeah in in quality wise besides that sliding window so all these besides that sliding window so all these besides that sliding window so all these are sub techniques which are sub techniques which are sub techniques which yeah yeah all these are some techniques yeah yeah all these are some techniques yeah yeah all these are some techniques like I just slide the windows all those like I just slide the windows all those like I just slide the windows all those things and instead of yeah instead of things and instead of yeah instead of things and instead of yeah instead of multiplying everything so linear multiplying everything so linear multiplying everything so linear attention is just saying that sum attention is just saying that sum attention is just saying that sum summarize everything first uh and then summarize everything first uh and then summarize everything first uh and then look up into it and then mamba this is look up into it and then mamba this is look up into it and then mamba this is just a state space model. Yeah, I can cover that. Okay.

I can cover that. Okay. Uh cool. Uh thank you T. Uh cool. Uh thank you T. Uh cool. Uh thank you T. So for the model like optimizations we So for the model like optimizations we So for the model like optimizations we also have like the two notebooks also have like the two notebooks also have like the two notebooks here. here. here. So there will be So there will be So there will be I have to go to this. Okay. Uh so for the quantization uh like Okay. Uh so for the quantization uh like the demo the demo the demo uh this is is this already run? No. Let uh this is is this already run? No. Let uh this is is this already run? No. Let me just run this. Okay. So we are loading the model which Okay. So we are loading the model which is like uh ML 7B. Wait.

Wait. Uh, did it run? Okay. So, it's uh two millisecond run. Okay. So, it's uh two millisecond run. Did this run? Okay. So, yeah, this time Did this run? Okay. So, yeah, this time Did this run? Okay. So, yeah, this time it's fetching that model with the FP16 it's fetching that model with the FP16 it's fetching that model with the FP16 precision. >> Yeah, it because it's downloading the >> Yeah, it because it's downloading the weights from the hugging face. weights from the hugging face. weights from the hugging face. >> Huh. Yeah. So, MOLAB is like running online. Yeah. So, MOLAB is like running online. >> Yes. because it needs to make the network because it needs to make the network call through to the hugging phase and call through to the hugging phase and call through to the hugging phase and like it fetching like it fetching like it fetching I don't know like but it's taking time I don't know like but it's taking time I don't know like but it's taking time to download probably So good. Okay. So here we see like the So good. Okay. So here we see like the memory size is like 15 GB around

memory size is like 15 GB around memory size is like 15 GB around approximately with the FP16 precision. approximately with the FP16 precision. approximately with the FP16 precision. We are trying to do the 2x compression We are trying to do the 2x compression We are trying to do the 2x compression as Tmet talked about with the int8. Let's download. Okay. So we do see like Let's download. Okay. So we do see like your memory size is now like 7.5 GB. your memory size is now like 7.5 GB. your memory size is now like 7.5 GB. What that means is now you have a more s What that means is now you have a more s What that means is now you have a more s more memory for your KV to basically more memory for your KV to basically more memory for your KV to basically grow. That means you can either serve grow. That means you can either serve grow. That means you can either serve higher context limit or you can serve higher context limit or you can serve higher context limit or you can serve the higher concurrent users there. If you do the like in your basic you are If you do the like in your basic you are doing the 4x compression so that with doing the 4x compression so that with doing the 4x compression so that with the 4x compression it would be more the 4x compression it would be more the 4x compression it would be more lower. It would be I think around lower. It would be I think around lower. It would be I think around 3 to 4 GB. 3 to 4 GB. 3 to 4 GB. Yeah. 4.5 GB and Yep. So this is Wait. and Yep. So this is Wait. So this is just a basic plot So this is just a basic plot So this is just a basic plot of like of like of like so these are the like theoretical so these are the like theoretical so these are the like theoretical numbers. uh we are not doing the like numbers. uh we are not doing the like numbers. uh we are not doing the like any throughput test here but uh usually any throughput test here but uh usually any throughput test here but uh usually you would see like your memory increases you would see like your memory increases you would see like your memory increases so pro you would also have like a bit of so pro you would also have like a bit of so pro you would also have like a bit of higher uh throughput.

higher uh throughput. higher uh throughput. Uh from some of the benchmarks that we Uh from some of the benchmarks that we Uh from some of the benchmarks that we studied we saw like the intate uh studied we saw like the intate uh studied we saw like the intate uh compression it does have like a lower compression it does have like a lower compression it does have like a lower throughput. Okay. And then there is like a demo on Okay. And then there is like a demo on the like the attention mechanisms. So for the attention okay I have to run So for the attention okay I have to run this. Uh okay so it has run. Oh, wait. Why Uh okay so it has run. Oh, wait. Why does it say no GPU detected? It should say the GPU should be It should say the GPU should be detected. Yeah, I guess it's not like able to

Yeah, I guess it's not like able to detect the GPU for some reason. Uh okay, never mind. Uh okay, never mind. Yeah. Yeah. So, but the like basic idea Yeah. Yeah. So, but the like basic idea Yeah. Yeah. So, but the like basic idea here was more like here was more like here was more like as you try to move towards like as you try to move towards like as you try to move towards like compressing the computation like by compressing the computation like by compressing the computation like by using different attention mechanisms using different attention mechanisms using different attention mechanisms like moving from the multi head to the like moving from the multi head to the like moving from the multi head to the grouped query attention and then to the grouped query attention and then to the grouped query attention and then to the MLA you would start seeing some MLA you would start seeing some MLA you would start seeing some optimizations. optimizations. optimizations. Um I think yesterday night we were doing Um I think yesterday night we were doing Um I think yesterday night we were doing some benchmarking. Uh I wanted to some benchmarking. Uh I wanted to some benchmarking. Uh I wanted to correct this part. Uh so it wasn't like correct this part. Uh so it wasn't like correct this part. Uh so it wasn't like 50 56x it was 14x. Uh basically the demo 50 56x it was 14x. Uh basically the demo 50 56x it was 14x. Uh basically the demo had a mistake of like a computation uh had a mistake of like a computation uh had a mistake of like a computation uh where it did not multiply the number of where it did not multiply the number of where it did not multiply the number of layers.

layers. layers. Uh yeah so apologies for that. Uh so Uh yeah so apologies for that. Uh so Uh yeah so apologies for that. Uh so this MLA is like a 14x savings work in this MLA is like a 14x savings work in this MLA is like a 14x savings work in comparison to like your multi head comparison to like your multi head comparison to like your multi head attention. Uh so now that we have understanding of Uh so now that we have understanding of the pain points, the foundations, the the pain points, the foundations, the the pain points, the foundations, the one side of the optimizations which is one side of the optimizations which is one side of the optimizations which is the model optimizations, the model optimizations, the model optimizations, we want to talk about what can you do on we want to talk about what can you do on we want to talk about what can you do on the like the serving side. the like the serving side. the like the serving side. So So So the first thing is we saw like when you the first thing is we saw like when you the first thing is we saw like when you perform like a simple decode step you perform like a simple decode step you perform like a simple decode step you are pulling it you are basically pulling are pulling it you are basically pulling are pulling it you are basically pulling the model weights and then you are the model weights and then you are the model weights and then you are recomputing the key and the value recomputing the key and the value recomputing the key and the value vectors for all the previous tokens even vectors for all the previous tokens even vectors for all the previous tokens even though you already computed the those though you already computed the those though you already computed the those vectors for the tokens. vectors for the tokens. vectors for the tokens. So there is definitely like a lot of So there is definitely like a lot of So there is definitely like a lot of compute wastage.

compute wastage. compute wastage. Uh and if you kind of analyze the time Uh and if you kind of analyze the time Uh and if you kind of analyze the time complexity of it, it would come out to complexity of it, it would come out to complexity of it, it would come out to be O of N². Uh and the way to resolve be O of N². Uh and the way to resolve be O of N². Uh and the way to resolve that is like a classic trade-off against that is like a classic trade-off against that is like a classic trade-off against the memory. You can maintain a memory of the memory. You can maintain a memory of the memory. You can maintain a memory of those vectors against the tokens and you those vectors against the tokens and you those vectors against the tokens and you can reference that memory. So that can reference that memory. So that can reference that memory. So that memory was called as like KV cache. memory was called as like KV cache. memory was called as like KV cache. uh and the like the flow looks something uh and the like the flow looks something uh and the like the flow looks something like this and then based on this KV cache there and then based on this KV cache there were like four optimizations that were were like four optimizations that were were like four optimizations that were really possible. really possible. really possible. Um Um Um the first one is about the page the first one is about the page the first one is about the page detention. So what's the different detention. So what's the different detention. So what's the different what's the problem today? So when you what's the problem today? So when you what's the problem today? So when you send like multiple requests as the input send like multiple requests as the input send like multiple requests as the input to the GPU to the GPU to the GPU these requests are in a batch these requests are in a batch these requests are in a batch uh every request is allocated like a uh every request is allocated like a uh every request is allocated like a continuous memory storage let's say of continuous memory storage let's say of continuous memory storage let's say of I'm just taking an example like let's I'm just taking an example like let's I'm just taking an example like let's set uh 2 KB set uh 2 KB set uh 2 KB however like your request needed only however like your request needed only however like your request needed only let's say let's say let's say uh 1 KB so there is like u 50% of that uh 1 KB so there is like u 50% of that uh 1 KB so there is like u 50% of that memory fragmentation.

memory fragmentation. memory fragmentation. Uh and this fragmentation basically Uh and this fragmentation basically Uh and this fragmentation basically leads to the memory wastage. That means leads to the memory wastage. That means leads to the memory wastage. That means there was a space in the memory where there was a space in the memory where there was a space in the memory where you could have served more requests but you could have served more requests but you could have served more requests but you could not because you were looking you could not because you were looking you could not because you were looking for that contigious block of the memory. for that contigious block of the memory. for that contigious block of the memory. So an inspiration to was being taken So an inspiration to was being taken So an inspiration to was being taken from like how the OS works like you from like how the OS works like you from like how the OS works like you maintain a logical memory and you maintain a logical memory and you maintain a logical memory and you basically have a physical memory. basically have a physical memory. basically have a physical memory. So in the logical memory it would still So in the logical memory it would still So in the logical memory it would still feel like feel like feel like uh that the KV vector for the like every uh that the KV vector for the like every uh that the KV vector for the like every token is like a contiguous token is like a contiguous token is like a contiguous but it will be mapping to a different but it will be mapping to a different but it will be mapping to a different physical address. So that really helped like saving a lot So that really helped like saving a lot of memory. Uh and it was only possible of memory. Uh and it was only possible of memory. Uh and it was only possible because you they considered like memory because you they considered like memory because you they considered like memory as a set of blocks and you would be as a set of blocks and you would be as a set of blocks and you would be dynamically allocating those blocks as dynamically allocating those blocks as dynamically allocating those blocks as the request need as the like new tokens the request need as the like new tokens the request need as the like new tokens comes in and they need that kind of comes in and they need that kind of comes in and they need that kind of memory.

The another lever is like when you are The another lever is like when you are sending multiple requests sending multiple requests sending multiple requests in the batch in the batch in the batch GPU is like taking those requests GPU is like taking those requests GPU is like taking those requests but it does not accepts the new batch but it does not accepts the new batch but it does not accepts the new batch unless all the requests in that batch unless all the requests in that batch unless all the requests in that batch gets completed. So the diagram looks gets completed. So the diagram looks gets completed. So the diagram looks more like a page retention but here it more like a page retention but here it more like a page retention but here it is more about like when is GPU available is more about like when is GPU available is more about like when is GPU available to take the next batch. So there is a to take the next batch. So there is a to take the next batch. So there is a time period where GPU is like sitting time period where GPU is like sitting time period where GPU is like sitting really idle really idle really idle and you want to like resolve for that and you want to like resolve for that and you want to like resolve for that and for that like the idea was like okay and for that like the idea was like okay and for that like the idea was like okay let's do that continuous batching. So the continuous batching also really So the continuous batching also really helped with like throughput because now helped with like throughput because now helped with like throughput because now you can ship more requests pretty you can ship more requests pretty you can ship more requests pretty quickly. Keep making sure like GPU quickly. Keep making sure like GPU quickly. Keep making sure like GPU always uh get is always like occupied always uh get is always like occupied always uh get is always like occupied and it's not like uh sitting idle. So and it's not like uh sitting idle. So and it's not like uh sitting idle. So you are saving on that compute.

you are saving on that compute. you are saving on that compute. The third is the like prefix caching. So The third is the like prefix caching. So The third is the like prefix caching. So you remember like the KV cache helped you remember like the KV cache helped you remember like the KV cache helped you save the computation for a single you save the computation for a single you save the computation for a single request across the tokens. request across the tokens. request across the tokens. But what if like you have the same But what if like you have the same But what if like you have the same tokens across multiple requests? How do tokens across multiple requests? How do tokens across multiple requests? How do you basically save against that? So the you basically save against that? So the you basically save against that? So the prefix caching uh which was introduced prefix caching uh which was introduced prefix caching uh which was introduced by VLM exactly counters that exactly counters that and then the third is like we talked and then the third is like we talked and then the third is like we talked about the fourth actually. So we talked about the fourth actually. So we talked about the fourth actually. So we talked about quantizing the model about quantizing the model about quantizing the model but you could also you can also like but you could also you can also like but you could also you can also like quantize the KV weights. quantize the KV weights. quantize the KV weights. So that means now you you need like a So that means now you you need like a So that means now you you need like a lesser space for your key and the value lesser space for your key and the value lesser space for your key and the value vectors. That means you can serve more vectors. That means you can serve more vectors. That means you can serve more key and the value vectors in the memory. key and the value vectors in the memory. key and the value vectors in the memory. And that means like you can serve more And that means like you can serve more And that means like you can serve more tokens. That means you can serve more tokens. That means you can serve more tokens. That means you can serve more context context limit. And that means context context limit. And that means context context limit. And that means like you can serve more model quality and all of this is like uh already and all of this is like uh already present in the VLM.

present in the VLM. present in the VLM. You don't really need to reinvent that You don't really need to reinvent that You don't really need to reinvent that wheel wheel wheel uh and you can like deploy this VLM in uh and you can like deploy this VLM in uh and you can like deploy this VLM in production and you could see that production and you could see that production and you could see that basically growth. basically growth. basically growth. So next we have like a benchmark that we So next we have like a benchmark that we So next we have like a benchmark that we did. So this benchmark was did. So this benchmark was did. So this benchmark was let me see if I have that let me see if I have that let me see if I have that here here here the demos. the demos. the demos. So doing this benchmark takes like So doing this benchmark takes like So doing this benchmark takes like around 1 hour because you have to around 1 hour because you have to around 1 hour because you have to continuously stop and like restart the continuously stop and like restart the continuously stop and like restart the VLM servers and you have to load the VLM servers and you have to load the VLM servers and you have to load the models and all. So it does take a lot of models and all. So it does take a lot of models and all. So it does take a lot of time in doing the testing but I can like time in doing the testing but I can like time in doing the testing but I can like really tell you here what we are doing. really tell you here what we are doing. really tell you here what we are doing. So we have kept the model as same like So we have kept the model as same like So we have kept the model as same like the Mistful 7B. the Mistful 7B. the Mistful 7B. Uh and then we have like the set of Uh and then we have like the set of Uh and then we have like the set of input questions that we are sending. Uh input questions that we are sending. Uh input questions that we are sending. Uh consider them as the prompts. Then we consider them as the prompts. Then we consider them as the prompts. Then we have couple of helper functions here have couple of helper functions here have couple of helper functions here like checking the server is up or not.

like checking the server is up or not. like checking the server is up or not. The server is the VLM server. The server is the VLM server. The server is the VLM server. Then there are helper functions to get Then there are helper functions to get Then there are helper functions to get the VLM metrics. the VLM metrics. the VLM metrics. uh and I will talk about like what those uh and I will talk about like what those uh and I will talk about like what those metrics are. Uh then there are like lot metrics are. Uh then there are like lot metrics are. Uh then there are like lot of the benchmarks and all and then you of the benchmarks and all and then you of the benchmarks and all and then you have to measure uh the KV usage and all. have to measure uh the KV usage and all. have to measure uh the KV usage and all. So these are the like helper functions. So these are the like helper functions. So these are the like helper functions. So the baseline is very simple like we So the baseline is very simple like we So the baseline is very simple like we have a hugging phase baseline. have a hugging phase baseline. have a hugging phase baseline. Uh this is the raw like sending the text Uh this is the raw like sending the text Uh this is the raw like sending the text to the LLM getting back the response. We to the LLM getting back the response. We to the LLM getting back the response. We see some results here. We saw like see some results here. We saw like see some results here. We saw like hugging phase has a throughput of like hugging phase has a throughput of like hugging phase has a throughput of like around 51 tokens per second. Time to around 51 tokens per second. Time to around 51 tokens per second. Time to first token was like 54 and then the first token was like 54 and then the first token was like 54 and then the inter token latency was 19. inter token latency was 19. inter token latency was 19. Uh this bas uh this was all run on the Uh this bas uh this was all run on the Uh this bas uh this was all run on the h100. h100. h100. Uh and then we start like a very default Uh and then we start like a very default Uh and then we start like a very default VLM server. So by default VLM provides VLM server. So by default VLM provides VLM server. So by default VLM provides you the page detention, continuous you the page detention, continuous you the page detention, continuous batching batching batching and the KV caching.

and the KV caching. and the KV caching. So three things are present by default and when you try to compare those and when you try to compare those benchmarks you see your throughput is benchmarks you see your throughput is benchmarks you see your throughput is like almost 15x you are able to serve like almost 15x you are able to serve like almost 15x you are able to serve more tokens per second more tokens per second more tokens per second then then then your time to the first token your time to the first token your time to the first token uh that also rises uh that also rises uh that also rises and then your v the inter token latency and then your v the inter token latency and then your v the inter token latency kind goes down and then your KV versus kind goes down and then your KV versus kind goes down and then your KV versus users and the versus context rate users and the versus context rate users and the versus context rate increases for sure. Now when you apply the prefix caching to Now when you apply the prefix caching to it it it so with the prefix caching you see like so with the prefix caching you see like so with the prefix caching you see like your throughput increases your throughput increases your throughput increases more your TDF decreases your inter token more your TDF decreases your inter token more your TDF decreases your inter token latency is approximately same uh and latency is approximately same uh and latency is approximately same uh and then your KV cache usage versus the then your KV cache usage versus the then your KV cache usage versus the users it's kind of going down users it's kind of going down users it's kind of going down the vers context it's not going down the vers context it's not going down the vers context it's not going down it's approximately same I think this is it's approximately same I think this is it's approximately same I think this is also approximately same it's like not also approximately same it's like not also approximately same it's like not that uh big of a deal that uh big of a deal that uh big of a deal when you apply the like KV quantization when you apply the like KV quantization when you apply the like KV quantization on top of it.

on top of it. on top of it. So it becomes like so so you see like So it becomes like so so you see like So it becomes like so so you see like your throughput is like almost similar. your throughput is like almost similar. your throughput is like almost similar. Your time to first token is similar. Your time to first token is similar. Your time to first token is similar. Your token latency is similar but then Your token latency is similar but then Your token latency is similar but then your KV usage actually goes down. And your KV usage actually goes down. And your KV usage actually goes down. And this is because like you have quantized this is because like you have quantized this is because like you have quantized your key value space. your key value space. your key value space. Uh and then there is a concept of Uh and then there is a concept of Uh and then there is a concept of speculative decoding that TME will talk speculative decoding that TME will talk speculative decoding that TME will talk about. Uh about. Uh about. Uh so when you try to benchmark those so so when you try to benchmark those so so when you try to benchmark those so you also see like there is a uh like a you also see like there is a uh like a you also see like there is a uh like a bit of like the less KV usage there bit of like the less KV usage there bit of like the less KV usage there although like the results are although like the results are although like the results are approximately same. So yeah, I mean overall like these are So yeah, I mean overall like these are the like the metrics across probably I the like the metrics across probably I the like the metrics across probably I should should should zoom out. Okay, it's not zoom out. It's zoom out. Okay, it's not zoom out. It's zoom out. Okay, it's not zoom out. It's not working.

not working. not working. Great. So yeah, this is the like VLM Great. So yeah, this is the like VLM Great. So yeah, this is the like VLM benchmarks. Um it's your production benchmarks. Um it's your production benchmarks. Um it's your production default by the way. uh we will also default by the way. uh we will also default by the way. uh we will also share that decision tree uh when we try share that decision tree uh when we try share that decision tree uh when we try to talk about like the other engines. So So yeah, so we should talk about like what yeah, so we should talk about like what yeah, so we should talk about like what are some of the other inference are some of the other inference are some of the other inference optimizations we can do on top of it and optimizations we can do on top of it and optimizations we can do on top of it and what were some of the other solutions what were some of the other solutions what were some of the other solutions that came out. Uh so I would like to again invite Uh so I would like to again invite Tanme. He's going to talk about like Tanme. He's going to talk about like Tanme. He's going to talk about like some of these optimizations. Oh, sorry. Uh I'm so sorry. Uh I didn't Oh, sorry. Uh I'm so sorry. Uh I didn't enable the slides. Uh what was the Okay, great. Uh what was the Okay, great. Perfect.

Perfect. Perfect. >> Which one? >> Which one? >> Which one? >> The speculative. >> The speculative. >> The speculative. >> Yeah. Thank you, Hersel. Yeah. >> Yeah. Thank you, Hersel. Yeah. >> Yeah. Thank you, Hersel. Yeah. So, So, So, so all these are like speculative so all these are like speculative so all these are like speculative decoding all these are the uh so so what decoding all these are the uh so so what decoding all these are the uh so so what we say uh different flavors of same kind we say uh different flavors of same kind we say uh different flavors of same kind of soda. So this uh this technique comes of soda. So this uh this technique comes of soda. So this uh this technique comes under decoding accelerator. So first one under decoding accelerator. So first one under decoding accelerator. So first one so we are only talking about this so we are only talking about this so we are only talking about this speculative decoding but there are other speculative decoding but there are other speculative decoding but there are other variants like self speculative eagle variants like self speculative eagle variants like self speculative eagle medusa medusa medusa I only like I think uh this one eagle I only like I think uh this one eagle I only like I think uh this one eagle algorithm algorithm algorithm personally I don't think personally I don't think personally I don't think speculative decoding works because main speculative decoding works because main speculative decoding works because main problem is alignment okay so let's start problem is alignment okay so let's start problem is alignment okay so let's start with what is uh speculative decoding with what is uh speculative decoding with what is uh speculative decoding main problem is that in transformer main problem is that in transformer main problem is that in transformer architecture All these tokens are architecture All these tokens are architecture All these tokens are generated sequentially one by one by generated sequentially one by one by generated sequentially one by one by one. How about just use a smaller model one. How about just use a smaller model one. How about just use a smaller model and let a smaller model to generate and let a smaller model to generate and let a smaller model to generate maybe let's say four or five tokens and maybe let's say four or five tokens and maybe let's say four or five tokens and this teacher model or we can say this teacher model or we can say this teacher model or we can say according to our world cup algorithm we according to our world cup algorithm we according to our world cup algorithm we can say referee. So referee will decide can say referee. So referee will decide can say referee. So referee will decide how many uh tokens it accept and this how many uh tokens it accept and this how many uh tokens it accept and this loop keeps on going on and our loop keeps on going on and our loop keeps on going on and our assumption is that there are certain assumption is that there are certain assumption is that there are certain domain where this kind of things will domain where this kind of things will domain where this kind of things will work like maybe in decode maybe in

work like maybe in decode maybe in work like maybe in decode maybe in coding or where almost there is no coding or where almost there is no coding or where almost there is no creativity uh each uh code or syntax is creativity uh each uh code or syntax is creativity uh each uh code or syntax is almost similar. So maybe it can help it. almost similar. So maybe it can help it. almost similar. So maybe it can help it. But uh based on personal testing, I But uh based on personal testing, I But uh based on personal testing, I didn't find this speculative decoding didn't find this speculative decoding didn't find this speculative decoding useful at all. But other techniques like useful at all. But other techniques like useful at all. But other techniques like uh self speculative decoding where uh self speculative decoding where uh self speculative decoding where teacher model also have one head teacher model also have one head teacher model also have one head auxiliary head and it will do same auxiliary head and it will do same auxiliary head and it will do same similar kind of things what this base similar kind of things what this base similar kind of things what this base model or small model is doing it. But model or small model is doing it. But model or small model is doing it. But then this eagle came Eagle 1 2 3 I don't then this eagle came Eagle 1 2 3 I don't then this eagle came Eagle 1 2 3 I don't know how many version versions are but know how many version versions are but know how many version versions are but it is just saying that instead of it is just saying that instead of it is just saying that instead of creating instead of generating tokens uh creating instead of generating tokens uh creating instead of generating tokens uh let's uh train a small model inside let's uh train a small model inside let's uh train a small model inside train a small model and just take a train a small model and just take a train a small model and just take a features from one of its uh one of main features from one of its uh one of main features from one of its uh one of main models layer so that instead of models layer so that instead of models layer so that instead of generating token uh it will generate uh generating token uh it will generate uh generating token uh it will generate uh this features so so uh so eagle is uh this features so so uh so eagle is uh this features so so uh so eagle is uh Eagle is better compared to this other Eagle is better compared to this other Eagle is better compared to this other kind of technologies and then another kind of technologies and then another kind of technologies and then another one is Medusa which is just saying that one is Medusa which is just saying that one is Medusa which is just saying that just generate all the tokens parallelly.

just generate all the tokens parallelly. just generate all the tokens parallelly. Uh okay. So here so here in this slide Uh okay. So here so here in this slide Uh okay. So here so here in this slide >> yeah the next slide. >> yeah the next slide. >> yeah the next slide. >> Okay. Okay. Yeah. Okay. Now we come to uh now Okay. Yeah. Okay. Now we come to uh now we will come to this one prefix caching. we will come to this one prefix caching. we will come to this one prefix caching. So I don't know whether people are using So I don't know whether people are using So I don't know whether people are using this one static prefix caching or not this one static prefix caching or not this one static prefix caching or not but thing is that main problem with but thing is that main problem with but thing is that main problem with prefix caching is that sometimes we type prefix caching is that sometimes we type prefix caching is that sometimes we type and make a small kind of mistake and and make a small kind of mistake and and make a small kind of mistake and this standard static prefix caching is this standard static prefix caching is this standard static prefix caching is basically it takes a prompt do some basically it takes a prompt do some basically it takes a prompt do some hashing and then next time when user hashing and then next time when user hashing and then next time when user asks similar kind of question it will asks similar kind of question it will asks similar kind of question it will try to match the hash. So if hash is uh try to match the hash. So if hash is uh try to match the hash. So if hash is uh if hash is equal then then it will if hash is equal then then it will if hash is equal then then it will instead of recomputing all those K and B instead of recomputing all those K and B instead of recomputing all those K and B it will just uh take it from from the it will just uh take it from from the it will just uh take it from from the storage but you know that sometimes we storage but you know that sometimes we storage but you know that sometimes we make a mistake or maybe we can just make a mistake or maybe we can just make a mistake or maybe we can just change a word or letter something like change a word or letter something like change a word or letter something like that then we have a very higher uh cache that then we have a very higher uh cache that then we have a very higher uh cache uh hit cache miss hit rate so that's why uh hit cache miss hit rate so that's why uh hit cache miss hit rate so that's why uh this one uh radics tree so radics uh this one uh radics tree so radics uh this one uh radics tree so radics tree is becoming very popular and also tree is becoming very popular and also tree is becoming very popular and also also because of agent. So I think almost also because of agent. So I think almost also because of agent. So I think almost everyone is doing agent and most of the everyone is doing agent and most of the everyone is doing agent and most of the computation is going during TT during computation is going during TT during computation is going during TT during test time inference kind of thing where test time inference kind of thing where test time inference kind of thing where we keep on asking same kind of questions we keep on asking same kind of questions we keep on asking same kind of questions and prompt for example you are an expert and prompt for example you are an expert and prompt for example you are an expert software engineer multiply by 200 times.

software engineer multiply by 200 times. software engineer multiply by 200 times. This kind of loop keeps on going inside This kind of loop keeps on going inside This kind of loop keeps on going inside this uh agentic agentic kind of things this uh agentic agentic kind of things this uh agentic agentic kind of things where it is al necessary to keep uh or where it is al necessary to keep uh or where it is al necessary to keep uh or store similar kind of things in a radics store similar kind of things in a radics store similar kind of things in a radics tree. So radics tree is just so so radic tree. So radics tree is just so so radic tree. So radics tree is just so so radic tree is just advanced version of this tree is just advanced version of this tree is just advanced version of this prefix tree where where we will just prefix tree where where we will just prefix tree where where we will just where we will just collapse a node if it where we will just collapse a node if it where we will just collapse a node if it does not have a does not have any branch does not have a does not have any branch does not have a does not have any branch and for this kind of work where keep on and for this kind of work where keep on and for this kind of work where keep on repeating same thing this uh red x tree repeating same thing this uh red x tree repeating same thing this uh red x tree helps a lot and st lang uh use this kind helps a lot and st lang uh use this kind helps a lot and st lang uh use this kind of algorithm of algorithm of algorithm for prefix caching. for prefix caching. for prefix caching. Okay. Yeah. Then there is another thing. Okay. Yeah. Then there is another thing. Okay. Yeah. Then there is another thing. One is tensor RT LLM. This is very One is tensor RT LLM. This is very One is tensor RT LLM. This is very confusing. When I first started, I was I confusing. When I first started, I was I confusing. When I first started, I was I was just confused. What is tensor RTLM?

was just confused. What is tensor RTLM? was just confused. What is tensor RTLM? So yeah. So tensor RT is just a uh it's So yeah. So tensor RT is just a uh it's So yeah. So tensor RT is just a uh it's just a standard uh SDK kind of thing. just a standard uh SDK kind of thing. just a standard uh SDK kind of thing. Tensor RTLM is just an inference engine Tensor RTLM is just an inference engine Tensor RTLM is just an inference engine just like VLM, SG lang. But problem is just like VLM, SG lang. But problem is just like VLM, SG lang. But problem is that it is related to Nvidia. They that it is related to Nvidia. They that it is related to Nvidia. They optimized each and every layer and every optimized each and every layer and every optimized each and every layer and every problem as I mentioned in our world cup problem as I mentioned in our world cup problem as I mentioned in our world cup algorithm. They just break everything algorithm. They just break everything algorithm. They just break everything and optimized everything at hardware and optimized everything at hardware and optimized everything at hardware level also. So uh yeah. So okay next. Yeah. So for this workshop we also uh Yeah. So for this workshop we also uh did some benchmarking like which is best did some benchmarking like which is best did some benchmarking like which is best uh so our setup was something similar uh so our setup was something similar uh so our setup was something similar was so so we did two kind of testing. was so so we did two kind of testing. was so so we did two kind of testing. First one is without uh without agentic First one is without uh without agentic First one is without uh without agentic testing where we just so testing where we just so testing where we just so we use this shared GPT uh this one data we use this shared GPT uh this one data we use this shared GPT uh this one data set and uh just ask those questions uh set and uh just ask those questions uh set and uh just ask those questions uh using VLM and SG lang.

using VLM and SG lang. using VLM and SG lang. Okay. Yeah. Okay. Yeah. Okay. And let me just zoom it up. Okay, great. And let me just zoom it up. Okay, great. And let me just zoom it up. Okay, great. Okay. Yeah. So, yeah, for this workshop, Okay. Yeah. So, yeah, for this workshop, Okay. Yeah. So, yeah, for this workshop, we used H100 and of our first testing we used H100 and of our first testing we used H100 and of our first testing was that uh we just uh we just asked uh was that uh we just uh we just asked uh was that uh we just uh we just asked uh we take questions from shared GPT and we take questions from shared GPT and we take questions from shared GPT and put it into VLM, SG lang and we found put it into VLM, SG lang and we found put it into VLM, SG lang and we found that actually there's no statistical that actually there's no statistical that actually there's no statistical difference between which one is better. difference between which one is better. difference between which one is better. So both have almost similar kind. So So both have almost similar kind. So So both have almost similar kind. So both are fulfilling similar kind of both are fulfilling similar kind of both are fulfilling similar kind of request per second uh TTFT and latency. request per second uh TTFT and latency. request per second uh TTFT and latency. So but only difference we have seen So but only difference we have seen So but only difference we have seen during agentic uh agentic branching. So during agentic uh agentic branching. So during agentic uh agentic branching. So uh what we did was that we asked that uh what we did was that we asked that uh what we did was that we asked that similar kind of question that you are similar kind of question that you are similar kind of question that you are the best this one software engineer in the best this one software engineer in the best this one software engineer in the world just solve the problem of the world just solve the problem of the world just solve the problem of traffic congestion in this city kind of traffic congestion in this city kind of traffic congestion in this city kind of thing. Then we put this into LLM. LLM thing. Then we put this into LLM. LLM thing. Then we put this into LLM. LLM generates some output. Then we did generates some output. Then we did generates some output. Then we did another uh round two also. So once this another uh round two also. So once this another uh round two also. So once this LLM generates this output, then in round LLM generates this output, then in round LLM generates this output, then in round two we have specially mentioned that uh two we have specially mentioned that uh two we have specially mentioned that uh provide uh review the proposal and give provide uh review the proposal and give provide uh review the proposal and give ratings from 1 to 10. So this uh two

ratings from 1 to 10. So this uh two ratings from 1 to 10. So this uh two turns we did uh and this loop keeps on turns we did uh and this loop keeps on turns we did uh and this loop keeps on uh repeating it. Uh what we found is uh repeating it. Uh what we found is uh repeating it. Uh what we found is that for this kind of that for this kind of that for this kind of uh workflow where everything is standard uh workflow where everything is standard uh workflow where everything is standard all those prompts and context all those prompts and context all those prompts and context engineering comes into the picture. If engineering comes into the picture. If engineering comes into the picture. If we do proper this agentic branching then we do proper this agentic branching then we do proper this agentic branching then I think uh this HG lang is three to four I think uh this HG lang is three to four I think uh this HG lang is three to four times better. But again this depends times better. But again this depends times better. But again this depends upon the different setup maybe uh if you upon the different setup maybe uh if you upon the different setup maybe uh if you do it uh you may get different results. do it uh you may get different results. do it uh you may get different results. Okay. Yeah. So I think uh did we Okay. Yeah. So I think uh did we Okay. Yeah. So I think uh did we uploaded it on GitHub? Okay. uploaded it on GitHub? Okay. uploaded it on GitHub? Okay. >> Yeah. So the PDF is like also in the >> Yeah. So the PDF is like also in the >> Yeah. So the PDF is like also in the drive. Uh it's the same link as the drive. Uh it's the same link as the drive. Uh it's the same link as the slides. slides. slides. So a quick summary here. So a quick summary here. So a quick summary here. So on a standard API workload throughput So on a standard API workload throughput So on a standard API workload throughput you would see like a VLM and the SG lang you would see like a VLM and the SG lang you would see like a VLM and the SG lang would behave same. So if you don't have would behave same. So if you don't have would behave same. So if you don't have if you have like a standard workload if you have like a standard workload if you have like a standard workload definitely go with VLM. It's the definitely go with VLM. It's the definitely go with VLM. It's the production default anyways. But what production default anyways. But what production default anyways. But what Tanme was also saying is when you try to Tanme was also saying is when you try to Tanme was also saying is when you try to like make it like agentic workloads that like make it like agentic workloads that like make it like agentic workloads that is where like your SG link really shines is where like your SG link really shines is where like your SG link really shines uh and uh it kind of like provides you uh and uh it kind of like provides you uh and uh it kind of like provides you all the benefits.

So yeah keep like VLM as a default but So yeah keep like VLM as a default but if you have agentic workloads probably if you have agentic workloads probably if you have agentic workloads probably try to move as the towards the SG lang. try to move as the towards the SG lang. try to move as the towards the SG lang. if you're not happy with DB LLM. Uh but if you're not happy with DB LLM. Uh but if you're not happy with DB LLM. Uh but uh okay. Uh let me And then like there is like the like a And then like there is like the like a comparison that is done at the 120 comparison that is done at the 120 comparison that is done at the 120 billion like for the GPTO OSS 120 billion like for the GPTO OSS 120 billion like for the GPTO OSS 120 billion. Um this is a benchmark that was billion. Um this is a benchmark that was billion. Um this is a benchmark that was prepared by clarify. So there is like a prepared by clarify. So there is like a prepared by clarify. So there is like a blog link here. Oh nice. blog link here. Oh nice. blog link here. Oh nice. Okay. Yeah. So they did the similar Okay. Yeah. So they did the similar Okay. Yeah. So they did the similar benchmark and they included like a benchmark and they included like a benchmark and they included like a tensor RT LLM in it. tensor RT LLM in it. tensor RT LLM in it. Definitely you can always go through Definitely you can always go through Definitely you can always go through these benchmarks and try to understand these benchmarks and try to understand these benchmarks and try to understand which basically suits your use case. As which basically suits your use case. As which basically suits your use case. As we mentioned like Tensor RT they try to we mentioned like Tensor RT they try to we mentioned like Tensor RT they try to optimize the hardware side as well optimize the hardware side as well optimize the hardware side as well having the peak hardware performance.

Yeah. Yeah. And then like in terms of when you want And then like in terms of when you want And then like in terms of when you want to dep pick like your engines to dep pick like your engines to dep pick like your engines once you figure out like between VLM, SG once you figure out like between VLM, SG once you figure out like between VLM, SG lang tenserati so that there are some lang tenserati so that there are some lang tenserati so that there are some new engines that are popping up Nvidia new engines that are popping up Nvidia new engines that are popping up Nvidia Dynamo for sure. Uh so they are also for Dynamo for sure. Uh so they are also for Dynamo for sure. Uh so they are also for the agentic uh session routing. the agentic uh session routing. the agentic uh session routing. Uh hugging phase is always there. It's a Uh hugging phase is always there. It's a Uh hugging phase is always there. It's a simple no server. Then there is like an simple no server. Then there is like an simple no server. Then there is like an MSAR MSAR MSAR engine that was recently proposed by engine that was recently proposed by engine that was recently proposed by Stanford. They are for like the Stanford. They are for like the Stanford. They are for like the multimodel. multimodel. multimodel. Uh so definitely you could explore those Uh so definitely you could explore those Uh so definitely you could explore those and when you try to basically just to and when you try to basically just to and when you try to basically just to like give a quick summary uh you we like give a quick summary uh you we like give a quick summary uh you we start with like a baseline start with like a baseline start with like a baseline we try to find what model could fit our we try to find what model could fit our we try to find what model could fit our use cases. use cases. use cases. Um, Um, Um, so you could pick like uh Deep Seek, you so you could pick like uh Deep Seek, you so you could pick like uh Deep Seek, you could pick like don't pick like a could pick like don't pick like a could pick like don't pick like a Mistral 7B. I mean, it's not good. Uh, Mistral 7B. I mean, it's not good. Uh, Mistral 7B. I mean, it's not good. Uh, but yeah, so you pick your model and you but yeah, so you pick your model and you but yeah, so you pick your model and you want to like have a smaller memory and want to like have a smaller memory and want to like have a smaller memory and you want to try to fit that bigger model you want to try to fit that bigger model you want to try to fit that bigger model into smaller memory so that you could into smaller memory so that you could into smaller memory so that you could save cost on the GPU cost. So you can do save cost on the GPU cost. So you can do save cost on the GPU cost. So you can do like all those com quantization like all those com quantization like all those com quantization then you can apply all those serving then you can apply all those serving then you can apply all those serving optimizations by using the right serving optimizations by using the right serving optimizations by using the right serving engine under the hood. So that can

engine under the hood. So that can engine under the hood. So that can really provide you that really provide you that really provide you that throughut that you really want. And And now something that you can do uh after now something that you can do uh after now something that you can do uh after going back home pro because we cannot going back home pro because we cannot going back home pro because we cannot like actually go over all the material like actually go over all the material like actually go over all the material here uh is definitely reading about some here uh is definitely reading about some here uh is definitely reading about some of the source informations like of the source informations like of the source informations like different attention mechanisms different different attention mechanisms different different attention mechanisms different like these engines like try to just read like these engines like try to just read like these engines like try to just read the different benchmarks which are the different benchmarks which are the different benchmarks which are present online as well present online as well present online as well and then there are a lot of like and then there are a lot of like and then there are a lot of like in-depth guides or the next phases of it in-depth guides or the next phases of it in-depth guides or the next phases of it which is like learning about some KV which is like learning about some KV which is like learning about some KV eviction strategies. So world is moving eviction strategies. So world is moving eviction strategies. So world is moving towards having a separate KV cache towards having a separate KV cache towards having a separate KV cache engineering domain. So you want to engineering domain. So you want to engineering domain. So you want to understand what's going on in there. So understand what's going on in there. So understand what's going on in there. So KV cache KV eviction cache compressions KV cache KV eviction cache compressions KV cache KV eviction cache compressions hybrid memories. So there are like lot hybrid memories. So there are like lot hybrid memories. So there are like lot of solutions that are happening around of solutions that are happening around of solutions that are happening around there. So always try to stick to those there. So always try to stick to those there. So always try to stick to those foundations or like the fundamentals or foundations or like the fundamentals or foundations or like the fundamentals or the first principles and try to see like the first principles and try to see like the first principles and try to see like which solution basically solves what which solution basically solves what which solution basically solves what problem and whether you actually need problem and whether you actually need problem and whether you actually need that problem to be solved for your use that problem to be solved for your use that problem to be solved for your use case case case and then there is like distributed LLM and then there is like distributed LLM and then there is like distributed LLM inference which is like a different inference which is like a different inference which is like a different painoint altogether. Uh you would painoint altogether. Uh you would painoint altogether. Uh you would probably need like a two-hour workshop probably need like a two-hour workshop probably need like a two-hour workshop there as well.

there as well. there as well. uh to like go over like all the uh to like go over like all the uh to like go over like all the internals do all the hands-on. Yes. And this is something we are trying Yes. And this is something we are trying to propose for the AI engineer New York to propose for the AI engineer New York to propose for the AI engineer New York session uh which is to like dive deeper session uh which is to like dive deeper session uh which is to like dive deeper into the advanced sections of the LLM into the advanced sections of the LLM into the advanced sections of the LLM inference. So this workshop was more for inference. So this workshop was more for inference. So this workshop was more for the like beginner and the intermediate the like beginner and the intermediate the like beginner and the intermediate level. Um so in this form we do have level. Um so in this form we do have level. Um so in this form we do have like a feedback as well plus also the like a feedback as well plus also the like a feedback as well plus also the interest. Um if uh you think like we interest. Um if uh you think like we interest. Um if uh you think like we need certain improvements on certain need certain improvements on certain need certain improvements on certain sections definitely give that feedback sections definitely give that feedback sections definitely give that feedback as well and if you want to see this as well and if you want to see this as well and if you want to see this workshop in like New York uh fair you I workshop in like New York uh fair you I workshop in like New York uh fair you I mean definitely feel free to enroll your mean definitely feel free to enroll your mean definitely feel free to enroll your interest. interest. interest. Uhhuh. Uhhuh. Uhhuh. How is it possible? How is it possible? How is it possible? Well, >> Huh?

>> Huh? >> URL works, right? Not the QR code. Okay. >> URL works, right? Not the QR code. Okay. >> URL works, right? Not the QR code. Okay. Probably I forgot to link those two Probably I forgot to link those two Probably I forgot to link those two together. Z Z G A five. Okay, cool. Yes. So if you can give that Okay, cool. Yes. So if you can give that feedback let me just okay that will be fine um and yeah I think we that will be fine um and yeah I think we would like to wrap this workshop then would like to wrap this workshop then would like to wrap this workshop then I'm sure like lot of you would be having I'm sure like lot of you would be having I'm sure like lot of you would be having a lot of questions so we can take all a lot of questions so we can take all a lot of questions so we can take all those like offline uh we can meet uh and those like offline uh we can meet uh and those like offline uh we can meet uh and we can uh like talk about those we can uh like talk about those we can uh like talk about those questions. questions. questions. >> Yeah sure. Uh thank you everyone. Thanks >> Yeah sure. Uh thank you everyone. Thanks >> Yeah sure. Uh thank you everyone. Thanks for joining. Uh I think it was really for joining. Uh I think it was really for joining. Uh I think it was really meaningful and all of you like came meaningful and all of you like came meaningful and all of you like came here. Uh thanks a lot.

here. Uh thanks a lot. here. Uh thanks a lot. >> Yeah. Thanks.