KV Cache-Aware Routing and P/D Disaggregation on Kubernetes — Yuchen Fama & Ashish Kamra, Red Hat

All right. Um, welcome everyone to yet All right. Um, welcome everyone to yet another inference talk. I hope you have another inference talk. I hope you have another inference talk. I hope you have had a good conference so far. And u, so had a good conference so far. And u, so had a good conference so far. And u, so in this session, I mean I'm sure you in this session, I mean I'm sure you in this session, I mean I'm sure you people who have been in the room uh must people who have been in the room uh must people who have been in the room uh must have heard these terms many times by have heard these terms many times by have heard these terms many times by now. So we're going to do a little bit now. So we're going to do a little bit now. So we're going to do a little bit more deep dive into the challenges of more deep dive into the challenges of more deep dive into the challenges of LLM deployments for agentic workloads LLM deployments for agentic workloads LLM deployments for agentic workloads and uh in this session we'll focus and uh in this session we'll focus and uh in this session we'll focus specifically on KV cache away routing specifically on KV cache away routing specifically on KV cache away routing and uh PD disagregation and uh PD disagregation and uh PD disagregation um and also you know when you when you um and also you know when you when you um and also you know when you when you look at public inference uh benchmark look at public inference uh benchmark look at public inference uh benchmark results you are typically looking at results you are typically looking at results you are typically looking at very steady state isolated highly very steady state isolated highly very steady state isolated highly sanitized numbers and what those sanitized numbers and what those sanitized numbers and what those benchmarks actually don't show you u is benchmarks actually don't show you u is benchmarks actually don't show you u is the chaotic reality of multi-turn the chaotic reality of multi-turn the chaotic reality of multi-turn interactions, massive context interactions, massive context interactions, massive context fluctuations which are very typical of fluctuations which are very typical of fluctuations which are very typical of agentic workloads. So we'll also try to agentic workloads. So we'll also try to agentic workloads. So we'll also try to pull the curtain back on some of those pull the curtain back on some of those pull the curtain back on some of those complexities. Um by by way of complexities. Um by by way of complexities. Um by by way of introduction uh my name is Ashish Kamra. introduction uh my name is Ashish Kamra. introduction uh my name is Ashish Kamra. I'm a senior manager of performance I'm a senior manager of performance I'm a senior manager of performance engineering at Red Hat. And with me engineering at Red Hat. And with me engineering at Red Hat. And with me >> hi I'm Yuch Chen. I'm the product >> hi I'm Yuch Chen. I'm the product >> hi I'm Yuch Chen. I'm the product manager at Red Hat Inference working manager at Red Hat Inference working manager at Red Hat Inference working closely with VLM and AMD core closely with VLM and AMD core closely with VLM and AMD core maintainers. also a contributor myself.

maintainers. also a contributor myself. maintainers. also a contributor myself. >> So here is the agenda for the next 20 >> So here is the agenda for the next 20 >> So here is the agenda for the next 20 minutes or so. Um Euchen will start with minutes or so. Um Euchen will start with minutes or so. Um Euchen will start with an analysis of inference behavior in the an analysis of inference behavior in the an analysis of inference behavior in the agentic era and some of the core agentic era and some of the core agentic era and some of the core characteristics and challenges. Uh next characteristics and challenges. Uh next characteristics and challenges. Uh next we next you will walk us through the KV we next you will walk us through the KV we next you will walk us through the KV cache um utilization and management cache um utilization and management cache um utilization and management strategies. strategies. strategies. I will break down the mechanics of I will break down the mechanics of I will break down the mechanics of pre-fill decode disagregation and walk pre-fill decode disagregation and walk pre-fill decode disagregation and walk you through some some results and then you through some some results and then you through some some results and then Euchen will again bring it all back Euchen will again bring it all back Euchen will again bring it all back together with our ongoing case study on together with our ongoing case study on together with our ongoing case study on our favorite open coding model GLM 5.2. our favorite open coding model GLM 5.2. our favorite open coding model GLM 5.2. Um and just a couple of uh sources from Um and just a couple of uh sources from Um and just a couple of uh sources from our side if you are more interested in our side if you are more interested in our side if you are more interested in learning more about open source learning more about open source learning more about open source inference we have a free course free inference we have a free course free inference we have a free course free course on deep learning.ai AI uh by course on deep learning.ai AI uh by course on deep learning.ai AI uh by Cedric and with Andrew Ning. Um and the Cedric and with Andrew Ning. Um and the Cedric and with Andrew Ning. Um and the other is a series of blogs on the Red other is a series of blogs on the Red other is a series of blogs on the Red Hat developer portal on distributed Hat developer portal on distributed Hat developer portal on distributed inference concepts uh troubleshooting inference concepts uh troubleshooting inference concepts uh troubleshooting and deployment patterns. and deployment patterns. and deployment patterns. Uh and for those who may not be aware Uh and for those who may not be aware Uh and for those who may not be aware since Red Hat is better known as the since Red Hat is better known as the since Red Hat is better known as the Linux company for enterprise Linux and Linux company for enterprise Linux and Linux company for enterprise Linux and uh the Kubernetes company for Open Shift uh the Kubernetes company for Open Shift uh the Kubernetes company for Open Shift uh but more recently we are also a major uh but more recently we are also a major uh but more recently we are also a major player in open source AI inference with player in open source AI inference with player in open source AI inference with uh us being the top contributor in VLM uh us being the top contributor in VLM uh us being the top contributor in VLM LLMD and the case of projects and also LLMD and the case of projects and also LLMD and the case of projects and also uh having incubated guide LLM for uh having incubated guide LLM for uh having incubated guide LLM for benchmarking LLM compressor for model benchmarking LLM compressor for model benchmarking LLM compressor for model quantization and speculators for uh

quantization and speculators for uh quantization and speculators for uh speculative uh decoding models and we speculative uh decoding models and we speculative uh decoding models and we also bring it bring all of that together also bring it bring all of that together also bring it bring all of that together in a optimized model hub on hugging in a optimized model hub on hugging in a optimized model hub on hugging phase under the Red Hat AI arc. phase under the Red Hat AI arc. phase under the Red Hat AI arc. Um and we are also building the platform Um and we are also building the platform Um and we are also building the platform for the next wave of agentic inference for the next wave of agentic inference for the next wave of agentic inference workloads and with that I will hand over workloads and with that I will hand over workloads and with that I will hand over to you to uh walk you through more of to you to uh walk you through more of to you to uh walk you through more of it. So we are currently um at this So we are currently um at this inflection point moving from the era of inflection point moving from the era of inflection point moving from the era of classic inference to the agentic era. So classic inference to the agentic era. So classic inference to the agentic era. So when we look at the real world agentic when we look at the real world agentic when we look at the real world agentic work workloads such as uh sweet bench work workloads such as uh sweet bench work workloads such as uh sweet bench and also watrices from real world cloud and also watrices from real world cloud and also watrices from real world cloud code sessions they fundamentally break code sessions they fundamentally break code sessions they fundamentally break many assumptions we made with classic LM many assumptions we made with classic LM many assumptions we made with classic LM serving. uh as you heard actually many serving. uh as you heard actually many serving. uh as you heard actually many times in previous sessions for example times in previous sessions for example times in previous sessions for example multi-turns and new standard we found multi-turns and new standard we found multi-turns and new standard we found from a few turns all the way to 3,000 from a few turns all the way to 3,000 from a few turns all the way to 3,000 turns and also because agent frequently turns and also because agent frequently turns and also because agent frequently reuse the uh system prompt and the total reuse the uh system prompt and the total reuse the uh system prompt and the total definitions we usually see super high definitions we usually see super high definitions we usually see super high cash hit rate um oftentimes well cash hit rate um oftentimes well cash hit rate um oftentimes well exceeding 90%. Uh another thing is input exceeding 90%. Uh another thing is input exceeding 90%. Uh another thing is input output ratio is uh is massive oftentimes output ratio is uh is massive oftentimes output ratio is uh is massive oftentimes over a 100 ratio and even higher and in over a 100 ratio and even higher and in over a 100 ratio and even higher and in many cases and on top of that the many cases and on top of that the many cases and on top of that the context management is is incredibly context management is is incredibly context management is is incredibly complex due to this high variance complex due to this high variance complex due to this high variance because we can't just simply take the because we can't just simply take the because we can't just simply take the average and oftentimes we need to look average and oftentimes we need to look average and oftentimes we need to look at the distributions and the P90 numbers at the distributions and the P90 numbers at the distributions and the P90 numbers especially when you do uh capacity

especially when you do uh capacity especially when you do uh capacity planning and also we observe really planning and also we observe really planning and also we observe really interesting patterns like sub Asian interesting patterns like sub Asian interesting patterns like sub Asian panel which is which further complex uh panel which is which further complex uh panel which is which further complex uh complicates scheduling. So to help complicates scheduling. So to help complicates scheduling. So to help communities study um this patterns we communities study um this patterns we communities study um this patterns we collaborate with Google thank you and collaborate with Google thank you and collaborate with Google thank you and also IBM our parent company to add uh a also IBM our parent company to add uh a also IBM our parent company to add uh a a trace replay tool in the inference a trace replay tool in the inference a trace replay tool in the inference perf you heard from earlier sessions u perf you heard from earlier sessions u perf you heard from earlier sessions u from Ashoken and Jason. Um so yeah feel from Ashoken and Jason. Um so yeah feel from Ashoken and Jason. Um so yeah feel free to check it out and the link is free to check it out and the link is free to check it out and the link is here. Uh next slide. Oh, so transition from Uh next slide. Oh, so transition from the class uh the characteristics um we the class uh the characteristics um we the class uh the characteristics um we just saw for agentic workloads. We're no just saw for agentic workloads. We're no just saw for agentic workloads. We're no longer chasing this um this this raw longer chasing this um this this raw longer chasing this um this this raw throughput in a steady state. We often throughput in a steady state. We often throughput in a steady state. We often need to optimize uh for example need to optimize uh for example need to optimize uh for example interactive latency and they're very um interactive latency and they're very um interactive latency and they're very um highly volatile and client-driven highly volatile and client-driven highly volatile and client-driven context because user and you know client context because user and you know client context because user and you know client define the prompt structure. So this define the prompt structure. So this define the prompt structure. So this introduced several critical challenges. introduced several critical challenges. introduced several critical challenges. First of all, KV cache management First of all, KV cache management First of all, KV cache management becomes super volatile because the becomes super volatile because the becomes super volatile because the context is client determined as I said.

context is client determined as I said. context is client determined as I said. So oftentimes we face this like you know So oftentimes we face this like you know So oftentimes we face this like you know frequent evictions and rewrites and frequent evictions and rewrites and frequent evictions and rewrites and secondly we also need to tune um the secondly we also need to tune um the secondly we also need to tune um the engine like VM with upper layer uh engine like VM with upper layer uh engine like VM with upper layer uh scheduling and routing. scheduling and routing. scheduling and routing. It needs that coordination such as It needs that coordination such as It needs that coordination such as prefix routing especially when latency prefix routing especially when latency prefix routing especially when latency becomes a primary uh scheduling matrix becomes a primary uh scheduling matrix becomes a primary uh scheduling matrix rather than like a secondary or rather than like a secondary or rather than like a secondary or afterthought. And thirdly, we also need afterthought. And thirdly, we also need afterthought. And thirdly, we also need to rethink our metrics. For example, we to rethink our metrics. For example, we to rethink our metrics. For example, we need to measure cats throughput need to measure cats throughput need to measure cats throughput separately. Why? Because on the right, separately. Why? Because on the right, separately. Why? Because on the right, it's really clear that economic stakes it's really clear that economic stakes it's really clear that economic stakes is very high. So, this is the uh is very high. So, this is the uh is very high. So, this is the uh anthropic API pricing. You also heard anthropic API pricing. You also heard anthropic API pricing. You also heard from earlier sessions. There's 10x cost from earlier sessions. There's 10x cost from earlier sessions. There's 10x cost difference between cash and non-cash difference between cash and non-cash difference between cash and non-cash tokens. So, 10x difference on your um tokens. So, 10x difference on your um tokens. So, 10x difference on your um token balance sheet is is pretty serious token balance sheet is is pretty serious token balance sheet is is pretty serious impact on your business. impact on your business. impact on your business. So next let's let's look at how the KV So next let's let's look at how the KV So next let's let's look at how the KV cache is um both utilized and managed in cache is um both utilized and managed in cache is um both utilized and managed in LMD. So LMD router has this really LMD. So LMD router has this really LMD. So LMD router has this really flexible um endpoint picker plugins we flexible um endpoint picker plugins we flexible um endpoint picker plugins we call the EP that can route the request call the EP that can route the request call the EP that can route the request to the optimal pods and that meet the KV to the optimal pods and that meet the KV to the optimal pods and that meet the KV cache locality and also the load cache locality and also the load cache locality and also the load criteria. So the EP continue probe each criteria. So the EP continue probe each criteria. So the EP continue probe each pods like VM pod matrix to score each pods like VM pod matrix to score each pods like VM pod matrix to score each pod on like the running for example pod on like the running for example pod on like the running for example running and waiting request and then the running and waiting request and then the running and waiting request and then the KV cache utilization also prefix uh KV cache utilization also prefix uh KV cache utilization also prefix uh cache availability and so we can cache availability and so we can cache availability and so we can schedule requests to the optimal pod schedule requests to the optimal pod schedule requests to the optimal pod with the lowest load and also highest with the lowest load and also highest with the lowest load and also highest possibility to um to of a cache hit. So possibility to um to of a cache hit. So possibility to um to of a cache hit. So um going down from to the KV cache

um going down from to the KV cache um going down from to the KV cache management layer actually you also heard management layer actually you also heard management layer actually you also heard from earlier session right before this. from earlier session right before this. from earlier session right before this. So for agentic sessions when you have u So for agentic sessions when you have u So for agentic sessions when you have u hot warm and cold cache our current hot warm and cold cache our current hot warm and cold cache our current effort focus on for example um more effort focus on for example um more effort focus on for example um more offloading tiers like NVME SSD and also offloading tiers like NVME SSD and also offloading tiers like NVME SSD and also uh file system XF along with KV ccentric uh file system XF along with KV ccentric uh file system XF along with KV ccentric store um like uh moon cake and also store um like uh moon cake and also store um like uh moon cake and also implementing smarter and session a wire implementing smarter and session a wire implementing smarter and session a wire eviction policies such as priority and eviction policies such as priority and eviction policies such as priority and also session pinning to uh ensure this also session pinning to uh ensure this also session pinning to uh ensure this uh really important you know the the uh really important you know the the uh really important you know the the context persists exactly when and where context persists exactly when and where context persists exactly when and where it's needed. So, I'm gonna play this um video really So, I'm gonna play this um video really quick. Uh it's a it's a short demo. quick. Uh it's a it's a short demo. quick. Uh it's a it's a short demo. >> Stand here so you can look at it. >> Stand here so you can look at it. >> Stand here so you can look at it. >> Okay. So, So, okay. So, this is a example of a KV okay. So, this is a example of a KV okay. So, this is a example of a KV cache bar routing. As you see, when we cache bar routing. As you see, when we cache bar routing. As you see, when we send the very first request and it send the very first request and it send the very first request and it populate the KV cache, it takes roughly populate the KV cache, it takes roughly populate the KV cache, it takes roughly 3 seconds. And when we actually look at 3 seconds. And when we actually look at 3 seconds. And when we actually look at where it's you know the KV cache uh is where it's you know the KV cache uh is where it's you know the KV cache uh is going there's no KV cache hit because going there's no KV cache hit because going there's no KV cache hit because it's the very first turn. And then when it's the very first turn. And then when it's the very first turn. And then when we have the second turn the request we have the second turn the request we have the second turn the request actually reuse a KV cache because as you actually reuse a KV cache because as you actually reuse a KV cache because as you see the system prompt is the same and see the system prompt is the same and see the system prompt is the same and this time takes about one seconds. And this time takes about one seconds. And this time takes about one seconds. And then when you actually look at the uh then when you actually look at the uh then when you actually look at the uh pod address exactly the same because we pod address exactly the same because we pod address exactly the same because we define the KV cache. Now going to the define the KV cache. Now going to the define the KV cache. Now going to the third turn a new request with different third turn a new request with different third turn a new request with different system prompt. Now it takes about three system prompt. Now it takes about three system prompt. Now it takes about three uh seconds and as you see you know right uh seconds and as you see you know right uh seconds and as you see you know right now and we don't find any KV cache here

now and we don't find any KV cache here now and we don't find any KV cache here because you can tell it's different pod because you can tell it's different pod because you can tell it's different pod address and then if you just change the address and then if you just change the address and then if you just change the user prompt and keep the same system user prompt and keep the same system user prompt and keep the same system prompt and the next turn you you reuse prompt and the next turn you you reuse prompt and the next turn you you reuse the KB cache and in this in this time it the KB cache and in this in this time it the KB cache and in this in this time it takes roughly about uh one second. Yeah. takes roughly about uh one second. Yeah. takes roughly about uh one second. Yeah. So it's a pretty intuitive demo and um So it's a pretty intuitive demo and um So it's a pretty intuitive demo and um I'll turn it to Ashish to talk about the I'll turn it to Ashish to talk about the I'll turn it to Ashish to talk about the next side but before that what does next side but before that what does next side but before that what does problem does it solve? So often times problem does it solve? So often times problem does it solve? So often times the prefix routing KB cache routing the prefix routing KB cache routing the prefix routing KB cache routing helps you solve the TTFD problem and of helps you solve the TTFD problem and of helps you solve the TTFD problem and of course you'll improve your lat uh your course you'll improve your lat uh your course you'll improve your lat uh your your throughput but oftentimes for your throughput but oftentimes for your throughput but oftentimes for agentic workload is not just a TTFT your agentic workload is not just a TTFT your agentic workload is not just a TTFT your throughput is about your inter token throughput is about your inter token throughput is about your inter token latency how do we solve that so preview latency how do we solve that so preview latency how do we solve that so preview decode disagregation is a really uh decode disagregation is a really uh decode disagregation is a really uh powerful technique but there are times powerful technique but there are times powerful technique but there are times there work at times it doesn't work so there work at times it doesn't work so there work at times it doesn't work so I'll turn it to Ashish to give you a I'll turn it to Ashish to give you a I'll turn it to Ashish to give you a preview of um of the PD uh disregation So before we dive into PD, let's just uh So before we dive into PD, let's just uh look at what LLMD is. So LLMD is a high look at what LLMD is. So LLMD is a high look at what LLMD is. So LLMD is a high performance Kubernetes native and performance Kubernetes native and performance Kubernetes native and actually now works on non-cubernetes actually now works on non-cubernetes actually now works on non-cubernetes environments as well. Distributed LM LLM environments as well. Distributed LM LLM environments as well. Distributed LM LLM inference framework hosted under the inference framework hosted under the inference framework hosted under the CNCF umbrella. LLMD provides a unified CNCF umbrella. LLMD provides a unified CNCF umbrella. LLMD provides a unified intelligent control plane designed intelligent control plane designed intelligent control plane designed specifically for agentic era of specifically for agentic era of specifically for agentic era of inference workloads. Well, Euchin inference workloads. Well, Euchin inference workloads. Well, Euchin already talked about the router and the already talked about the router and the already talked about the router and the EP at the top of the slide. Um, the EP at the top of the slide. Um, the EP at the top of the slide. Um, the other aspects are workload APIs such as other aspects are workload APIs such as other aspects are workload APIs such as leader worker set and disagregated set leader worker set and disagregated set leader worker set and disagregated set that orchestrates complex multi- that orchestrates complex multi- that orchestrates complex multi- multi-node model execution and then

multi-node model execution and then multi-node model execution and then autoscalers that monitors capacity autoscalers that monitors capacity autoscalers that monitors capacity bounds and real-time traffic mixes to bounds and real-time traffic mixes to bounds and real-time traffic mixes to independently scale up and scale down uh independently scale up and scale down uh independently scale up and scale down uh your pods depending on the system load. your pods depending on the system load. your pods depending on the system load. So now look now let's look at uh prefill So now look now let's look at uh prefill So now look now let's look at uh prefill decode disagregation in detail. Um uh decode disagregation in detail. Um uh decode disagregation in detail. Um uh okay so why does PD exist in the first okay so why does PD exist in the first okay so why does PD exist in the first place? So one of the most powerful place? So one of the most powerful place? So one of the most powerful patterns implemented by LLMD is prefill patterns implemented by LLMD is prefill patterns implemented by LLMD is prefill decode disagregation and you must have decode disagregation and you must have decode disagregation and you must have heard from some of the previous talks as heard from some of the previous talks as heard from some of the previous talks as well. So what happens is in in a nonPD well. So what happens is in in a nonPD well. So what happens is in in a nonPD situation in aggregated serving one pod situation in aggregated serving one pod situation in aggregated serving one pod is responsible for optimizing both your is responsible for optimizing both your is responsible for optimizing both your time to first token and your inter token time to first token and your inter token time to first token and your inter token latencies. Uh but in PD prefill and latencies. Uh but in PD prefill and latencies. Uh but in PD prefill and decode become independently scalable decode become independently scalable decode become independently scalable inference pods. But to understand why we inference pods. But to understand why we inference pods. But to understand why we actually need this we have to look at actually need this we have to look at actually need this we have to look at the physics of LLM execution. the physics of LLM execution. the physics of LLM execution. colloccating uh both prefill and decode colloccating uh both prefill and decode colloccating uh both prefill and decode tasks on the same GPU creates something tasks on the same GPU creates something tasks on the same GPU creates something called as phase interference. Prefill called as phase interference. Prefill called as phase interference. Prefill phase is the phase that creates the KV phase is the phase that creates the KV phase is the phase that creates the KV caches for your initial prompt. It wants caches for your initial prompt. It wants caches for your initial prompt. It wants high compute. It's highly bursty uh high compute. It's highly bursty uh high compute. It's highly bursty uh utilizes GPUs at uh high flops and and utilizes GPUs at uh high flops and and utilizes GPUs at uh high flops and and thrives on large batch parallelism to thrives on large batch parallelism to thrives on large batch parallelism to process the prompts and builds the process the prompts and builds the process the prompts and builds the initial KV cache. The decode phase on initial KV cache. The decode phase on initial KV cache. The decode phase on the other hand is generating one token the other hand is generating one token the other hand is generating one token at a time and it's more me memory at a time and it's more me memory at a time and it's more me memory bandwidth hungry. It's highly latency bandwidth hungry. It's highly latency bandwidth hungry. It's highly latency sensitive and requires high heavy cache sensitive and requires high heavy cache sensitive and requires high heavy cache residency. So in a in a in a traditional

residency. So in a in a in a traditional residency. So in a in a in a traditional aggregated pod if you if there's a aggregated pod if you if there's a aggregated pod if you if there's a sudden influx of a long prefilled palm, sudden influx of a long prefilled palm, sudden influx of a long prefilled palm, it will completely stall the ongoing it will completely stall the ongoing it will completely stall the ongoing decode token generation process causing decode token generation process causing decode token generation process causing massive problems and jitter in user massive problems and jitter in user massive problems and jitter in user streaming latency. streaming latency. streaming latency. So, so how does PD actually work in So, so how does PD actually work in So, so how does PD actually work in practice in LMD? So, LNMD uses um uh you practice in LMD? So, LNMD uses um uh you practice in LMD? So, LNMD uses um uh you know like okay, we'll start with step know like okay, we'll start with step know like okay, we'll start with step one. A incoming request hits the gateway one. A incoming request hits the gateway one. A incoming request hits the gateway router which dynamically evaluates router which dynamically evaluates router which dynamically evaluates cluster states using something known as cluster states using something known as cluster states using something known as the endpoint picker you talked about and the endpoint picker you talked about and the endpoint picker you talked about and schedules the request to use PD schedules the request to use PD schedules the request to use PD disagregation selecting the optimal disagregation selecting the optimal disagregation selecting the optimal prefill and decode workers. The router prefill and decode workers. The router prefill and decode workers. The router then coordinates the transaction then coordinates the transaction then coordinates the transaction directly with the designated pre-fill directly with the designated pre-fill directly with the designated pre-fill worker. The pre-fill worker processes worker. The pre-fill worker processes worker. The pre-fill worker processes the prompt, construct the initial KV the prompt, construct the initial KV the prompt, construct the initial KV cache of the prompt and outputs the cache of the prompt and outputs the cache of the prompt and outputs the standard KV transfer metadata. Um, and standard KV transfer metadata. Um, and standard KV transfer metadata. Um, and the target decode worker actually pulls the target decode worker actually pulls the target decode worker actually pulls the computed KV caches um, uh, across the computed KV caches um, uh, across the computed KV caches um, uh, across the network fab fabric utilizing uh, the the network fab fabric utilizing uh, the the network fab fabric utilizing uh, the KV transfer metadata that the uh, uh, KV transfer metadata that the uh, uh, KV transfer metadata that the uh, uh, prefill pod had generated. Um okay so prefill pod had generated. Um okay so prefill pod had generated. Um okay so with that yes that's kind of how uh PD with that yes that's kind of how uh PD with that yes that's kind of how uh PD is implemented in practice in LMD and is implemented in practice in LMD and is implemented in practice in LMD and next I would like to show you some uh next I would like to show you some uh next I would like to show you some uh experimental results on where PD experimental results on where PD experimental results on where PD actually shines. So in this graph you actually shines. So in this graph you actually shines. So in this graph you can see that um can see that um can see that um uh in in in the standard aggregated uh in in in the standard aggregated uh in in in the standard aggregated deployment which is the top red line uh

deployment which is the top red line uh deployment which is the top red line uh the P99 ITL uh hovers roughly around 900 the P99 ITL uh hovers roughly around 900 the P99 ITL uh hovers roughly around 900 milliseconds and you can you can see milliseconds and you can you can see milliseconds and you can you can see some fluctuations um up and down and but some fluctuations um up and down and but some fluctuations um up and down and but the the bottom blue line is the P99 uh the the bottom blue line is the P99 uh the the bottom blue line is the P99 uh inter token latency on a PD deployment inter token latency on a PD deployment inter token latency on a PD deployment and you can see that it's drastically and you can see that it's drastically and you can see that it's drastically almost nine times better at 100 almost nine times better at 100 almost nine times better at 100 millconds and it's also much smoother uh millconds and it's also much smoother uh millconds and it's also much smoother uh than the aggregated serving and uh this is some of our own internal and uh this is some of our own internal results at Red Hat. So for a GPOSS 12B results at Red Hat. So for a GPOSS 12B results at Red Hat. So for a GPOSS 12B model uh 16 H100s model uh 16 H100s model uh 16 H100s uh the aggregated config is four uh the aggregated config is four uh the aggregated config is four replicas tensor parallelism 4 and the replicas tensor parallelism 4 and the replicas tensor parallelism 4 and the disagregated is two prefilled 2D code disagregated is two prefilled 2D code disagregated is two prefilled 2D code all with tensor parallelism 4. It's a all with tensor parallelism 4. It's a all with tensor parallelism 4. It's a highly multi-turn workload with a 10,000 highly multi-turn workload with a 10,000 highly multi-turn workload with a 10,000 token prefix and 128 tokens for every token prefix and 128 tokens for every token prefix and 128 tokens for every turn every turn. So, so this is a great turn every turn. So, so this is a great turn every turn. So, so this is a great chart like you can see at the bottom chart like you can see at the bottom chart like you can see at the bottom most line is a standard aggregated most line is a standard aggregated most line is a standard aggregated config that's uh is doing the default config that's uh is doing the default config that's uh is doing the default Kubernetes scheduling and uh and it's Kubernetes scheduling and uh and it's Kubernetes scheduling and uh and it's aggregated. So that's kind of our aggregated. So that's kind of our aggregated. So that's kind of our baseline and then the middle blue line baseline and then the middle blue line baseline and then the middle blue line is still aggregated but with the LLMD uh is still aggregated but with the LLMD uh is still aggregated but with the LLMD uh KV cache aware routing and you can KV cache aware routing and you can KV cache aware routing and you can almost see the gains just just based on almost see the gains just just based on almost see the gains just just based on the routing and the red line is actually the routing and the red line is actually the routing and the red line is actually the PD uh the pre-fill decode config the PD uh the pre-fill decode config the PD uh the pre-fill decode config with two pre-fill and two decode workers with two pre-fill and two decode workers with two pre-fill and two decode workers and you can actually see that like it's and you can actually see that like it's and you can actually see that like it's very similar to the aggregated config at very similar to the aggregated config at very similar to the aggregated config at the lower concurrency regimes and uh

the lower concurrency regimes and uh the lower concurrency regimes and uh even and and very similar at the higher even and and very similar at the higher even and and very similar at the higher concurrency regimes but it's actually concurrency regimes but it's actually concurrency regimes but it's actually the middle part of the concurrency the middle part of the concurrency the middle part of the concurrency regime that PD actually shines regime that PD actually shines regime that PD actually shines and and these are some of the the and and these are some of the the and and these are some of the the classic parita curves that we see when classic parita curves that we see when classic parita curves that we see when you actually do PD and uh aggregated you actually do PD and uh aggregated you actually do PD and uh aggregated side by side. So these results are again side by side. So these results are again side by side. So these results are again from the GPTOSS 12B model 64 H100s from the GPTOSS 12B model 64 H100s from the GPTOSS 12B model 64 H100s aggregated is eight replicas TP8 and aggregated is eight replicas TP8 and aggregated is eight replicas TP8 and this a is uh three prefilled 5D code this a is uh three prefilled 5D code this a is uh three prefilled 5D code again TP8 and a pre-filled heavy again TP8 and a pre-filled heavy again TP8 and a pre-filled heavy workload with like 5,000 average input workload with like 5,000 average input workload with like 5,000 average input sequence length and 500 output sequence sequence length and 500 output sequence sequence length and 500 output sequence length and you can actually see the blue length and you can actually see the blue length and you can actually see the blue line is the the PD curve and the red line is the the PD curve and the red line is the the PD curve and the red line is the aggregated curve and the PD line is the aggregated curve and the PD line is the aggregated curve and the PD curve kind of dominates um uh the curve kind of dominates um uh the curve kind of dominates um uh the aggregate curve across the entire aggregate curve across the entire aggregate curve across the entire interactivity spectrum. Okay, but I don't want to leave you guys Okay, but I don't want to leave you guys that PD is the answer to everything and that PD is the answer to everything and that PD is the answer to everything and it's a magic bullet. But um it's uh it's it's a magic bullet. But um it's uh it's it's a magic bullet. But um it's uh it's essentially a separation phase essentially a separation phase essentially a separation phase separation trade-off and not a magic separation trade-off and not a magic separation trade-off and not a magic bullet. So we created this uh matrix to bullet. So we created this uh matrix to bullet. So we created this uh matrix to help you decide when PD might be uh good help you decide when PD might be uh good help you decide when PD might be uh good for you. So if you're managing long for you. So if you're managing long for you. So if you're managing long context uh with high ISL OSL ratios and context uh with high ISL OSL ratios and context uh with high ISL OSL ratios and you if you have a large model that you if you have a large model that you if you have a large model that you're serving that can that you can you're serving that can that you can you're serving that can that you can apply rich model parallelism techniques apply rich model parallelism techniques apply rich model parallelism techniques um you're facing that middle concurrency um you're facing that middle concurrency um you're facing that middle concurrency regime uh that I I showed you in the regime uh that I I showed you in the regime uh that I I showed you in the previous graphs and and the very

previous graphs and and the very previous graphs and and the very important part is that if you want uh important part is that if you want uh important part is that if you want uh strict ITL streaming requirements like strict ITL streaming requirements like strict ITL streaming requirements like you want the you want the token you want the you want the token you want the you want the token generation to be uh much more smooth um generation to be uh much more smooth um generation to be uh much more smooth um then you want to consider PD but we also then you want to consider PD but we also then you want to consider PD but we also saw that it requires transfer of KV saw that it requires transfer of KV saw that it requires transfer of KV caches from your pre-filled workers to caches from your pre-filled workers to caches from your pre-filled workers to your decode workers. So you must pro your decode workers. So you must pro your decode workers. So you must pro process an advanced uh high-sp speeded process an advanced uh high-sp speeded process an advanced uh high-sp speeded network fabric like uh RDMMA or rocky to network fabric like uh RDMMA or rocky to network fabric like uh RDMMA or rocky to support that KV cache transfer. And if support that KV cache transfer. And if support that KV cache transfer. And if you do not have such requirements, short you do not have such requirements, short you do not have such requirements, short moderate context, any model size, low moderate context, any model size, low moderate context, any model size, low concurrency regimes or uh if you have concurrency regimes or uh if you have concurrency regimes or uh if you have strict TTF requirements because you can strict TTF requirements because you can strict TTF requirements because you can actually tune them on an aggregate actually tune them on an aggregate actually tune them on an aggregate serving um and you the biggest point is serving um and you the biggest point is serving um and you the biggest point is like if you don't have the network like if you don't have the network like if you don't have the network fabric to support those KV cache fabric to support those KV cache fabric to support those KV cache transfers. So you might actually just transfers. So you might actually just transfers. So you might actually just want to stick with aggregated. want to stick with aggregated. want to stick with aggregated. So here is my key takeaway from all of So here is my key takeaway from all of So here is my key takeaway from all of this. So architecting this complex this. So architecting this complex this. So architecting this complex platform requires balancing a lot of u platform requires balancing a lot of u platform requires balancing a lot of u knobs and a highly multi-dimensional knobs and a highly multi-dimensional knobs and a highly multi-dimensional design space all of which is supported design space all of which is supported design space all of which is supported in LLMD. As you saw the scheduler must in LLMD. As you saw the scheduler must in LLMD. As you saw the scheduler must support or constantly evaluate SLO support or constantly evaluate SLO support or constantly evaluate SLO targets uh QEPs KV cache locality targets uh QEPs KV cache locality targets uh QEPs KV cache locality metrics PD ratios and network topologies metrics PD ratios and network topologies metrics PD ratios and network topologies to be able to route the request to the to be able to route the request to the to be able to route the request to the optimal FOD. While in while the PD optimal FOD. While in while the PD optimal FOD. While in while the PD design space you you need dynamic PD design space you you need dynamic PD design space you you need dynamic PD rate matching to adapt to PD ratios rate matching to adapt to PD ratios rate matching to adapt to PD ratios because you know you can start with a because you know you can start with a because you know you can start with a static PD ratio but it needs to evolve static PD ratio but it needs to evolve static PD ratio but it needs to evolve with the autoscaler as the traffic with the autoscaler as the traffic with the autoscaler as the traffic changes um and you need uh yeah

changes um and you need uh yeah changes um and you need uh yeah autoscaling to scale PD pools autoscaling to scale PD pools autoscaling to scale PD pools independently independently independently um and constantly tweaking model um and constantly tweaking model um and constantly tweaking model parallelism techniques like tensor parallelism techniques like tensor parallelism techniques like tensor parallelism data parallelism uh to meet parallelism data parallelism uh to meet parallelism data parallelism uh to meet your SLOs's. your SLOs's. your SLOs's. So um I think with these uh I will hand So um I think with these uh I will hand So um I think with these uh I will hand it over to Euchen to anchor some of the it over to Euchen to anchor some of the it over to Euchen to anchor some of the concepts that we showed with the real concepts that we showed with the real concepts that we showed with the real world case study of serving the GLM 5.2 world case study of serving the GLM 5.2 world case study of serving the GLM 5.2 model uh which is uh still ongoing as we model uh which is uh still ongoing as we model uh which is uh still ongoing as we speak. speak. speak. >> Yeah, still ongoing. You probably have >> Yeah, still ongoing. You probably have >> Yeah, still ongoing. You probably have seen tons of uh impressive numbers of seen tons of uh impressive numbers of seen tons of uh impressive numbers of GLM 5.2 on B200 when we talk to our GLM 5.2 on B200 when we talk to our GLM 5.2 on B200 when we talk to our customers and they usually don't have customers and they usually don't have customers and they usually don't have you know the luxury of B200. They have a you know the luxury of B200. They have a you know the luxury of B200. They have a lot of H200. So we have to figure out lot of H200. So we have to figure out lot of H200. So we have to figure out how to like put all the knobs together how to like put all the knobs together how to like put all the knobs together and make GM 5.2 work really well for and make GM 5.2 work really well for and make GM 5.2 work really well for cluster of of H200. So uh we let's cluster of of H200. So uh we let's cluster of of H200. So uh we let's anchor all the concept together. Um we anchor all the concept together. Um we anchor all the concept together. Um we went through for example the uh KV cache went through for example the uh KV cache went through for example the uh KV cache routing PD disagregation. We kind of routing PD disagregation. We kind of routing PD disagregation. We kind of call them a wildl path in LMD and also call them a wildl path in LMD and also call them a wildl path in LMD and also we combine with different parallelism we combine with different parallelism we combine with different parallelism strategies to so we can uh independently strategies to so we can uh independently strategies to so we can uh independently uh scale prefuel paths because for uh scale prefuel paths because for uh scale prefuel paths because for agentic workload is super uh long you agentic workload is super uh long you agentic workload is super uh long you know like heavy prefill. So uh in this know like heavy prefill. So uh in this know like heavy prefill. So uh in this case we designed the prefuel pool using case we designed the prefuel pool using case we designed the prefuel pool using up to three workers optimized for uh up to three workers optimized for uh up to three workers optimized for uh high throughput uh with deep and then high throughput uh with deep and then high throughput uh with deep and then for decoup we use uh one dedicated for decoup we use uh one dedicated for decoup we use uh one dedicated worker and um that's optimized for for worker and um that's optimized for for worker and um that's optimized for for low latency. So we use Nixo for low latency. So we use Nixo for low latency. So we use Nixo for efficient KV transfer between the pools efficient KV transfer between the pools efficient KV transfer between the pools and also with the each worker we have and also with the each worker we have and also with the each worker we have the leader worker set group uh with TP1

the leader worker set group uh with TP1 the leader worker set group uh with TP1 DP8 and also uh EP8 uh expert DP8 and also uh EP8 uh expert DP8 and also uh EP8 uh expert parallelism 8. So the architecture is parallelism 8. So the architecture is parallelism 8. So the architecture is just highly modular because you can uh just highly modular because you can uh just highly modular because you can uh actually scale the throughput by simply actually scale the throughput by simply actually scale the throughput by simply adding uh preview workers without adding uh preview workers without adding uh preview workers without reconfiguring and um the decoup. So uh reconfiguring and um the decoup. So uh reconfiguring and um the decoup. So uh this highlights how AMD effectly this highlights how AMD effectly this highlights how AMD effectly effectively managed the complexity of effectively managed the complexity of effectively managed the complexity of combining like PB and DB and EPI scale. combining like PB and DB and EPI scale. combining like PB and DB and EPI scale. And also we found some interesting fun And also we found some interesting fun And also we found some interesting fun fact actually a couple days ago. Um B B fact actually a couple days ago. Um B B fact actually a couple days ago. Um B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B B Bf6 B B B B B B B B B B B B B B B B B B Bf6 B B B B B B B B B B B B B B B B B B Bf6 uh BF16 KV cache actually is faster than uh BF16 KV cache actually is faster than uh BF16 KV cache actually is faster than using like FPA uh KV cache for longer using like FPA uh KV cache for longer using like FPA uh KV cache for longer preview. Um this is also like we preview. Um this is also like we preview. Um this is also like we continue to explore and found like more continue to explore and found like more continue to explore and found like more interesting patterns, but more interesting patterns, but more interesting patterns, but more importantly uh we want to kind of just importantly uh we want to kind of just importantly uh we want to kind of just show the result really quick. So um for show the result really quick. So um for show the result really quick. So um for this uh data set agentic workload data this uh data set agentic workload data this uh data set agentic workload data set the ISO OSL ratio is pretty high 45 set the ISO OSL ratio is pretty high 45 set the ISO OSL ratio is pretty high 45 to1 ratio preview is uh is really the to1 ratio preview is uh is really the to1 ratio preview is uh is really the constraint you can tell um with 2P even constraint you can tell um with 2P even constraint you can tell um with 2P even 1D we have um 4x passer TDFT and also 60 1D we have um 4x passer TDFT and also 60 1D we have um 4x passer TDFT and also 60 uh% more requests and this is continuous uh% more requests and this is continuous uh% more requests and this is continuous like work in progress so the next step like work in progress so the next step like work in progress so the next step is we need to also put the upper layer is we need to also put the upper layer is we need to also put the upper layer lower TTFT and also adding more more lower TTFT and also adding more more lower TTFT and also adding more more preview replicas so um I know we're preview replicas so um I know we're preview replicas so um I know we're running out of time really quick. Uh we running out of time really quick. Uh we running out of time really quick. Uh we um the fundamental shift for agentic um the fundamental shift for agentic um the fundamental shift for agentic workload we're continuing to uh have workload we're continuing to uh have workload we're continuing to uh have this um uh agentic north uh northstar uh this um uh agentic north uh northstar uh this um uh agentic north uh northstar uh with session graph orchestration program with session graph orchestration program with session graph orchestration program award scheduling uh state reuse life award scheduling uh state reuse life award scheduling uh state reuse life cycle and also the uh agentic benchmark cycle and also the uh agentic benchmark cycle and also the uh agentic benchmark um we're working on. So you can find

um we're working on. So you can find um we're working on. So you can find them uh in AMD upstream AMD and also you them uh in AMD upstream AMD and also you them uh in AMD upstream AMD and also you know feel free to join the SIG group and know feel free to join the SIG group and know feel free to join the SIG group and uh and contribute and um this is the uh and contribute and um this is the uh and contribute and um this is the very last slide. So distri distributed very last slide. So distri distributed very last slide. So distri distributed inference is not challenge uh every inference is not challenge uh every inference is not challenge uh every single comp a single company can solve single comp a single company can solve single comp a single company can solve along. We're proud to be uh building along. We're proud to be uh building along. We're proud to be uh building this uh future in the open alongside our this uh future in the open alongside our this uh future in the open alongside our incredible ecosystem collaborators uh incredible ecosystem collaborators uh incredible ecosystem collaborators uh core wave Google IBM Nvidia growing list core wave Google IBM Nvidia growing list core wave Google IBM Nvidia growing list of launch partners and industry of launch partners and industry of launch partners and industry adopters. So if you're passionate about adopters. So if you're passionate about adopters. So if you're passionate about the future of opensource inference, we the future of opensource inference, we the future of opensource inference, we invite you to join us. We do have a invite you to join us. We do have a invite you to join us. We do have a booth downstairs. Feel free to stop by, booth downstairs. Feel free to stop by, booth downstairs. Feel free to stop by, ask us any questions. And uh thank you ask us any questions. And uh thank you ask us any questions. And uh thank you so much for your time. so much for your time. so much for your time. [applause]