Routing LLM Inference in Production: From Engine Signals to Policy — Qianru Lao & Lu Zhang, OpenAI

Hi everyone, thanks for joining our Hi everyone, thanks for joining our talk. I'm Lou and this is my colleague talk. I'm Lou and this is my colleague talk. I'm Lou and this is my colleague Chenu. So we today we're going to talk Chenu. So we today we're going to talk Chenu. So we today we're going to talk about the in uh we both work on the about the in uh we both work on the about the in uh we both work on the inference team at OpenAI and today we inference team at OpenAI and today we inference team at OpenAI and today we are going to talk about routing IM are going to talk about routing IM are going to talk about routing IM inference in production specifically how inference in production specifically how inference in production specifically how our system evolved from routing based on our system evolved from routing based on our system evolved from routing based on feedback loops driven by engine signals feedback loops driven by engine signals feedback loops driven by engine signals to a more explicit and a predictable to a more explicit and a predictable to a more explicit and a predictable policy which is still informed by engine policy which is still informed by engine policy which is still informed by engine signals. However, it's more like the way signals. However, it's more like the way signals. However, it's more like the way we use it is different. we use it is different. we use it is different. So uh for the agenda today we're going So uh for the agenda today we're going So uh for the agenda today we're going to begin by introducing the inference to begin by introducing the inference to begin by introducing the inference load balancer what it is what it does load balancer what it is what it does load balancer what it is what it does and how it has evolved and then Chenu and how it has evolved and then Chenu and how it has evolved and then Chenu will walk us through the newer control will walk us through the newer control will walk us through the newer control plane and data plane driven architecture plane and data plane driven architecture plane and data plane driven architecture uh what are the responsibilities of each uh what are the responsibilities of each uh what are the responsibilities of each and followed by a concrete case study of and followed by a concrete case study of and followed by a concrete case study of how we reduce the global network how we reduce the global network how we reduce the global network overhead and in the end I return to overhead and in the end I return to overhead and in the end I return to discuss the protection mechanisms that discuss the protection mechanisms that discuss the protection mechanisms that help keep the system stable and help keep the system stable and help keep the system stable and production level stress.

production level stress. production level stress. So to begin with what is the inference So to begin with what is the inference So to begin with what is the inference load balancer and where it sit? So this load balancer and where it sit? So this load balancer and where it sit? So this is a very high level diagram of the is a very high level diagram of the is a very high level diagram of the system we are talking about. On the left system we are talking about. On the left system we are talking about. On the left hand side are the front end clusters. hand side are the front end clusters. hand side are the front end clusters. Those are the GPU cluster. Sorry, those Those are the GPU cluster. Sorry, those Those are the GPU cluster. Sorry, those are the CPU clusters that act as are the CPU clusters that act as are the CPU clusters that act as gateways into our system and they gateways into our system and they gateways into our system and they receive user requests then prepare them receive user requests then prepare them receive user requests then prepare them into the inference request that can be into the inference request that can be into the inference request that can be processed by the inference engines and processed by the inference engines and processed by the inference engines and on the right hand side are the engine on the right hand side are the engine on the right hand side are the engine clusters uh which are usually GPU clusters uh which are usually GPU clusters uh which are usually GPU clusters and each hosting multiple clusters and each hosting multiple clusters and each hosting multiple inference engines. So that's why we they inference engines. So that's why we they inference engines. So that's why we they got the name of engine clusters and as got the name of engine clusters and as got the name of engine clusters and as you may already heard nowadays GPUs are you may already heard nowadays GPUs are you may already heard nowadays GPUs are pretty popular and expensive. So um pretty popular and expensive. So um pretty popular and expensive. So um sitting in the middle it is the IRB or sitting in the middle it is the IRB or sitting in the middle it is the IRB or inference load balancer. It actually inference load balancer. It actually inference load balancer. It actually runs on the front end clusters but is runs on the front end clusters but is runs on the front end clusters but is also a bridge into our inference stack.

also a bridge into our inference stack. also a bridge into our inference stack. It has two main responsibilities select It has two main responsibilities select It has two main responsibilities select an engine and proxing the request. For an engine and proxing the request. For an engine and proxing the request. For this talk, we are going to focus on the this talk, we are going to focus on the this talk, we are going to focus on the engine selection part. So in some ways, engine selection part. So in some ways, engine selection part. So in some ways, IRB resembles a very traditional load IRB resembles a very traditional load IRB resembles a very traditional load balancer because a request usually balancer because a request usually balancer because a request usually targets a model and a model is backed by targets a model and a model is backed by targets a model and a model is backed by multiple engines. They may live on multiple engines. They may live on multiple engines. They may live on different clusters in different regions different clusters in different regions different clusters in different regions or even across the continents because or even across the continents because or even across the continents because that gives us a good resiliency towards that gives us a good resiliency towards that gives us a good resiliency towards localized degradation or cluster localized degradation or cluster localized degradation or cluster failures. However, the inference uh failures. However, the inference uh failures. However, the inference uh stack or the uniqueness of the inference stack or the uniqueness of the inference stack or the uniqueness of the inference introduces a lot of nuances like uh it introduces a lot of nuances like uh it introduces a lot of nuances like uh it has to consider a bunch of signals has to consider a bunch of signals has to consider a bunch of signals reported in real time like the reported in real time like the reported in real time like the well-known time to first token TTF time well-known time to first token TTF time well-known time to first token TTF time between output tokens also known as between output tokens also known as between output tokens also known as token throughput or time between tokens token throughput or time between tokens token throughput or time between tokens and other hairness and utilization and other hairness and utilization and other hairness and utilization signals. Besides there's a important signals. Besides there's a important signals. Besides there's a important concept of a KV cache which is also well concept of a KV cache which is also well concept of a KV cache which is also well known but for example when the known but for example when the known but for example when the conversation already has a lot of the conversation already has a lot of the conversation already has a lot of the useful context cached in one engine useful context cached in one engine useful context cached in one engine sending the follow-up turns of the same sending the follow-up turns of the same sending the follow-up turns of the same conversation back to the same engine conversation back to the same engine conversation back to the same engine will avoid recomputation improve will avoid recomputation improve will avoid recomputation improve efficiency and uh reduce latency. So the efficiency and uh reduce latency. So the efficiency and uh reduce latency. So the combination of performance reliability combination of performance reliability combination of performance reliability locality cach awareness is what makes it locality cach awareness is what makes it locality cach awareness is what makes it such an interesting problem. Uh so how such an interesting problem. Uh so how such an interesting problem. Uh so how we attempted in the problem? Let's take we attempted in the problem? Let's take we attempted in the problem? Let's take a look at the early days. And to be

a look at the early days. And to be a look at the early days. And to be honest, early days in in this industry honest, early days in in this industry honest, early days in in this industry sounds a lot more historic than it sounds a lot more historic than it sounds a lot more historic than it really is. And the routing process at really is. And the routing process at really is. And the routing process at that time began with a fear of like each that time began with a fear of like each that time began with a fear of like each request may not be served by all the request may not be served by all the request may not be served by all the engines because of uh constraints such engines because of uh constraints such engines because of uh constraints such as capabilities or due uh restrictions as capabilities or due uh restrictions as capabilities or due uh restrictions due to compute or data residency. And due to compute or data residency. And due to compute or data residency. And among the remaining engines, IRB used a among the remaining engines, IRB used a among the remaining engines, IRB used a weighted consistent hashing to select weighted consistent hashing to select weighted consistent hashing to select the best destination engine for a the best destination engine for a the best destination engine for a request or of for certain user. Then the request or of for certain user. Then the request or of for certain user. Then the important question becomes where are the important question becomes where are the important question becomes where are the WS come from. So they were generated by WS come from. So they were generated by WS come from. So they were generated by a periodic feedback loop. The inference a periodic feedback loop. The inference a periodic feedback loop. The inference engines as mentioned earlier uh reports engines as mentioned earlier uh reports engines as mentioned earlier uh reports all kind of the signals we care about all kind of the signals we care about all kind of the signals we care about and the controller will periodically and the controller will periodically and the controller will periodically smooth out those signals and compute a smooth out those signals and compute a smooth out those signals and compute a performance score. The performance score performance score. The performance score performance score. The performance score then will be compared against the fleet then will be compared against the fleet then will be compared against the fleet average. Then the weight will be average. Then the weight will be average. Then the weight will be adjusted basically for each engine. is adjusted basically for each engine. is adjusted basically for each engine. is weight goes up if the performance is weight goes up if the performance is weight goes up if the performance is better or it goes down when the better or it goes down when the better or it goes down when the performance is worse than the fleet performance is worse than the fleet performance is worse than the fleet average and this generated weight will average and this generated weight will average and this generated weight will impact the routing and then it's impact the routing and then it's impact the routing and then it's basically a uh control loop um basically a uh control loop um basically a uh control loop um conceptually it is very similar to the P conceptually it is very similar to the P conceptually it is very similar to the P controller and no this P controller will controller and no this P controller will controller and no this P controller will not help you care a Linux process but not help you care a Linux process but not help you care a Linux process but instead it's a classic control theory

instead it's a classic control theory instead it's a classic control theory technique that continuously steering the technique that continuously steering the technique that continuously steering the system towards its desired date and we system towards its desired date and we system towards its desired date and we just borrowed this important concept the just borrowed this important concept the just borrowed this important concept the proportional part of it and uh applied proportional part of it and uh applied proportional part of it and uh applied into our uh our load balancer. So it has into our uh our load balancer. So it has into our uh our load balancer. So it has a lot of nice properties. For example, a lot of nice properties. For example, a lot of nice properties. For example, it could combine the useful signals we it could combine the useful signals we it could combine the useful signals we care about into the single routing care about into the single routing care about into the single routing decision and because of the it adapt to decision and because of the it adapt to decision and because of the it adapt to the observed performance as what we the observed performance as what we the observed performance as what we mentioned earlier there's a lot of mentioned earlier there's a lot of mentioned earlier there's a lot of constraints and those constraints might constraints and those constraints might constraints and those constraints might have the some engines basier because have the some engines basier because have the some engines basier because they can serve more requests more kind they can serve more requests more kind they can serve more requests more kind of request than the remaining but those of request than the remaining but those of request than the remaining but those basier signal will be fit into the next basier signal will be fit into the next basier signal will be fit into the next loop and resulting in the less loop and resulting in the less loop and resulting in the less constraint less constraint request can constraint less constraint request can constraint less constraint request can go to more of those kind of engines. So go to more of those kind of engines. So go to more of those kind of engines. So basically they self balanced out and to basically they self balanced out and to basically they self balanced out and to some extent this just means we don't some extent this just means we don't some extent this just means we don't need to do a lot of manual intervention need to do a lot of manual intervention need to do a lot of manual intervention and it just works. However that kind of and it just works. However that kind of and it just works. However that kind of adaptability comes with big trade-offs adaptability comes with big trade-offs adaptability comes with big trade-offs because of the same reasons that it because of the same reasons that it because of the same reasons that it combined so many signals. It's also very combined so many signals. It's also very combined so many signals. It's also very hard to reason about a particular hard to reason about a particular hard to reason about a particular routing decision or like why search routing decision or like why search routing decision or like why search engine get a higher weight than we engine get a higher weight than we engine get a higher weight than we expect. And every time we want to expect. And every time we want to expect. And every time we want to fine-tune towards some aspect, it's all fine-tune towards some aspect, it's all fine-tune towards some aspect, it's all almost impossible to not impacting almost impossible to not impacting almost impossible to not impacting something else.

something else. something else. And the load is not always very well And the load is not always very well And the load is not always very well very evenly distributed because uh very evenly distributed because uh very evenly distributed because uh sometimes a model is served by engines sometimes a model is served by engines sometimes a model is served by engines on different GPU skills and they have on different GPU skills and they have on different GPU skills and they have different characteristics. Then the different characteristics. Then the different characteristics. Then the problem becomes a lot more trickier and problem becomes a lot more trickier and problem becomes a lot more trickier and the feedback loop sometimes creates bad the feedback loop sometimes creates bad the feedback loop sometimes creates bad oscillations because when you shift an oscillations because when you shift an oscillations because when you shift an engine away engine away engine away some traffic the engine turns a bit some traffic the engine turns a bit some traffic the engine turns a bit cooler and this signal get fit to the cooler and this signal get fit to the cooler and this signal get fit to the controller. The controller now thinks controller. The controller now thinks controller. The controller now thinks that hey this engine can take a lot more that hey this engine can take a lot more that hey this engine can take a lot more traffic. then the some traffic going to traffic. then the some traffic going to traffic. then the some traffic going to be shifted back and forth between a few be shifted back and forth between a few be shifted back and forth between a few engines and disrupting the KV cache engines and disrupting the KV cache engines and disrupting the KV cache utilization. So all those limitations utilization. So all those limitations utilization. So all those limitations motivated us to rethink about the motivated us to rethink about the motivated us to rethink about the architecture and uh see if we have new architecture and uh see if we have new architecture and uh see if we have new ways to address the problem. So I'm ways to address the problem. So I'm ways to address the problem. So I'm going to hand over to Chenu to uh deep going to hand over to Chenu to uh deep going to hand over to Chenu to uh deep dive into the new architecture we tried dive into the new architecture we tried dive into the new architecture we tried out.

>> Yeah, thank you L. So I'm going to talk >> Yeah, thank you L. So I'm going to talk about the architecture of the load about the architecture of the load about the architecture of the load balancer and how do we reduce the balancer and how do we reduce the balancer and how do we reduce the overall overhead with our routing overall overhead with our routing overall overhead with our routing algorithm. The load balancer answers one algorithm. The load balancer answers one algorithm. The load balancer answers one question question question uh for each request from a CPU cluster uh for each request from a CPU cluster uh for each request from a CPU cluster which engine should serve it. One most which engine should serve it. One most which engine should serve it. One most naive baseline might be round robin naive baseline might be round robin naive baseline might be round robin which send requests across engines which send requests across engines which send requests across engines evenly. But if you think a little bit evenly. But if you think a little bit evenly. But if you think a little bit more that doesn't make sense because more that doesn't make sense because more that doesn't make sense because engines are not homogeneous they can engines are not homogeneous they can engines are not homogeneous they can have different hardware and capacity have different hardware and capacity have different hardware and capacity different health and also different different health and also different different health and also different distance from CPU cluster also run could distance from CPU cluster also run could distance from CPU cluster also run could break cache locality break cache locality break cache locality c related requests that could reuse the c related requests that could reuse the c related requests that could reuse the same engine cache might be sent to same engine cache might be sent to same engine cache might be sent to different engines. different engines. different engines. A probably better solution might be for A probably better solution might be for A probably better solution might be for each CPU cluster it choose the best each CPU cluster it choose the best each CPU cluster it choose the best engine from its own local view. But engine from its own local view. But engine from its own local view. But that's not enough either. Think about that's not enough either. Think about that's not enough either. Think about one extreme case. Multiple CPU cluster one extreme case. Multiple CPU cluster one extreme case. Multiple CPU cluster route traffic to the same engines route traffic to the same engines route traffic to the same engines independently which could overload that independently which could overload that independently which could overload that engine while leave other engines engine while leave other engines engine while leave other engines underutilized.

underutilized. underutilized. So what we need is a globally optimized So what we need is a globally optimized So what we need is a globally optimized solution, a control plane that has a solution, a control plane that has a solution, a control plane that has a global view for all the CPU cluster and global view for all the CPU cluster and global view for all the CPU cluster and GPU engines and could compute a globally GPU engines and could compute a globally GPU engines and could compute a globally optimized routing answers and the data optimized routing answers and the data optimized routing answers and the data plane can make a routing decision plane can make a routing decision plane can make a routing decision quickly based on the answer proved from quickly based on the answer proved from quickly based on the answer proved from the control plane. Now let's look inside the control plane Now let's look inside the control plane and data plane. In the data plane there and data plane. In the data plane there and data plane. In the data plane there is an engine selector which select is an engine selector which select is an engine selector which select engine for each request. It read the engine for each request. It read the engine for each request. It read the local routing state which includes the local routing state which includes the local routing state which includes the candidate engines and the routing candidate engines and the routing candidate engines and the routing weights for each candidate engines. Both weights for each candidate engines. Both weights for each candidate engines. Both of them are refreshed asynchronously in of them are refreshed asynchronously in of them are refreshed asynchronously in the background. So we don't need to ask the background. So we don't need to ask the background. So we don't need to ask the control plane before we make a the control plane before we make a the control plane before we make a routing decision for each request. Also routing decision for each request. Also routing decision for each request. Also the data plane collects realtime engine the data plane collects realtime engine the data plane collects realtime engine signal such as number of ready replica signal such as number of ready replica signal such as number of ready replica engine house etc to surface fast local engine house etc to surface fast local engine house etc to surface fast local guard draw guard draw guard draw in the control plane. The data loader in the control plane. The data loader in the control plane. The data loader combines those live engine signals and combines those live engine signals and combines those live engine signals and never overhead. And with offline never overhead. And with offline never overhead. And with offline regressions of capacity, TTFT and TBOT, regressions of capacity, TTFT and TBOT, regressions of capacity, TTFT and TBOT, the optimizer could turn those data into the optimizer could turn those data into the optimizer could turn those data into routing weights and the control plane routing weights and the control plane routing weights and the control plane will publish the routing weight for each will publish the routing weight for each will publish the routing weight for each data plane to pull. In this way, no data plane to pull. In this way, no data plane to pull. In this way, no request need to wait on the data plane.

request need to wait on the data plane. request need to wait on the data plane. The control plane continuously compute The control plane continuously compute The control plane continuously compute the next globally optimized routing way the next globally optimized routing way the next globally optimized routing way snapshot while the data plane make a snapshot while the data plane make a snapshot while the data plane make a routing decision based on the latest routing decision based on the latest routing decision based on the latest snapshot already installed locally. In summary, there are three important In summary, there are three important paths through the system. The first path paths through the system. The first path paths through the system. The first path is the inference request path. The is the inference request path. The is the inference request path. The request arrive to the CPU cluster and request arrive to the CPU cluster and request arrive to the CPU cluster and the data plane inside that CPU cluster the data plane inside that CPU cluster the data plane inside that CPU cluster will select engine for that request will select engine for that request will select engine for that request based on the local routing state and based on the local routing state and based on the local routing state and forward the request to the selected forward the request to the selected forward the request to the selected engines. engines. engines. The second pass is the engine signal The second pass is the engine signal The second pass is the engine signal pass. The system continuously collects pass. The system continuously collects pass. The system continuously collects real-time engine signal such as TTFT, real-time engine signal such as TTFT, real-time engine signal such as TTFT, TBOT, number of radio replica and engine TBOT, number of radio replica and engine TBOT, number of radio replica and engine house etc. Boost planes need those house etc. Boost planes need those house etc. Boost planes need those real-time engine signals. The control real-time engine signals. The control real-time engine signals. The control plane need them to compute a globally plane need them to compute a globally plane need them to compute a globally optimized routing way while the data optimized routing way while the data optimized routing way while the data plane need them to serve as fast local.

plane need them to serve as fast local. plane need them to serve as fast local. And the third path is the routing way And the third path is the routing way And the third path is the routing way pass. The control plane compute and pass. The control plane compute and pass. The control plane compute and publish the routing way and the data publish the routing way and the data publish the routing way and the data plane pull the updates to its local plane pull the updates to its local plane pull the updates to its local cache. cache. cache. So only the first pass is synchronous So only the first pass is synchronous So only the first pass is synchronous but it's fast and only local inside the but it's fast and only local inside the but it's fast and only local inside the data plane of the CPU cluster. The other data plane of the CPU cluster. The other data plane of the CPU cluster. The other two loops are asynchronous loop and they two loops are asynchronous loop and they two loops are asynchronous loop and they are to improve future routing decision. are to improve future routing decision. are to improve future routing decision. So that's pretty much of the So that's pretty much of the So that's pretty much of the architecture part. But that still leaves architecture part. But that still leaves architecture part. But that still leaves one question. How do we compute those one question. How do we compute those one question. How do we compute those routing weights? But before answer that question, let's But before answer that question, let's answer another question first. Why not answer another question first. Why not answer another question first. Why not just send a request to the nearest just send a request to the nearest just send a request to the nearest engine? engine? engine? That's because the traffic demand and That's because the traffic demand and That's because the traffic demand and GPU capacity are not geographically GPU capacity are not geographically GPU capacity are not geographically balanced. balanced. balanced. For example, in region one, CPU cluster For example, in region one, CPU cluster For example, in region one, CPU cluster A send 90 RPS and the nearby engine A A send 90 RPS and the nearby engine A A send 90 RPS and the nearby engine A can serve 100 RPS. So in this case can serve 100 RPS. So in this case can serve 100 RPS. So in this case nearest only is fine while in region two nearest only is fine while in region two nearest only is fine while in region two CPU cluster B send 120 RPS and the CPU cluster B send 120 RPS and the CPU cluster B send 120 RPS and the nearby engine B could only serve 100 nearby engine B could only serve 100 nearby engine B could only serve 100 RPS. So in this case if we insist on RPS. So in this case if we insist on RPS. So in this case if we insist on keeping everything local the extra 20 keeping everything local the extra 20 keeping everything local the extra 20 RPS need to wait on an overloaded engine RPS need to wait on an overloaded engine RPS need to wait on an overloaded engine B. While in region three we are only

B. While in region three we are only B. While in region three we are only using 40 RPS of an 80 RPS engine C. That using 40 RPS of an 80 RPS engine C. That using 40 RPS of an 80 RPS engine C. That still leaves 40 RPS spare. So if we send still leaves 40 RPS spare. So if we send still leaves 40 RPS spare. So if we send the extra 20 RPS from cluster B to the extra 20 RPS from cluster B to the extra 20 RPS from cluster B to engine C, that will add network engine C, that will add network engine C, that will add network distance. distance. distance. But it could also avoid a probably much But it could also avoid a probably much But it could also avoid a probably much larger engine side waiting time. So in larger engine side waiting time. So in larger engine side waiting time. So in this case, a further engine might be this case, a further engine might be this case, a further engine might be faster end to end. That's why we need faster end to end. That's why we need faster end to end. That's why we need something better than the nearest only something better than the nearest only something better than the nearest only routing. Now let's open the black box of the Now let's open the black box of the optimizer. optimizer. optimizer. The optimizer accepts four types of The optimizer accepts four types of The optimizer accepts four types of input. The request from each CPU input. The request from each CPU input. The request from each CPU cluster, the network latency to each cluster, the network latency to each cluster, the network latency to each engine, the available engine capacity engine, the available engine capacity engine, the available engine capacity and health and also the TTFT, TBOT and health and also the TTFT, TBOT and health and also the TTFT, TBOT latency profiles that tell us how's the latency profiles that tell us how's the latency profiles that tell us how's the engine side latency change as the low engine side latency change as the low engine side latency change as the low increases.

increases. increases. And with those input the optimizer turn And with those input the optimizer turn And with those input the optimizer turn the input to the output routing weights. the input to the output routing weights. the input to the output routing weights. The routing way say for each CPU cluster The routing way say for each CPU cluster The routing way say for each CPU cluster what fraction of its traffic should go what fraction of its traffic should go what fraction of its traffic should go to each GPU engine. to each GPU engine. to each GPU engine. And the optimization goal is And the optimization goal is And the optimization goal is straightforward is to minimize the straightforward is to minimize the straightforward is to minimize the expected end to end latency across all expected end to end latency across all expected end to end latency across all routed traffic. The important part is routed traffic. The important part is routed traffic. The important part is that the end to end latency includes that the end to end latency includes that the end to end latency includes both the network distance and the engine both the network distance and the engine both the network distance and the engine side latency. That means a nearby engine side latency. That means a nearby engine side latency. That means a nearby engine might be attractive when it still has might be attractive when it still has might be attractive when it still has room to serve traffic while a further room to serve traffic while a further room to serve traffic while a further engine might be better if all the nearby engine might be better if all the nearby engine might be better if all the nearby engines are close to full. engines are close to full. engines are close to full. And the optimizer also need to respect And the optimizer also need to respect And the optimizer also need to respect several hard constraints. First it need several hard constraints. First it need several hard constraints. First it need to route all the traffic demand. Second to route all the traffic demand. Second to route all the traffic demand. Second it need to ensure all the engines stay it need to ensure all the engines stay it need to ensure all the engines stay within the effected capacity. Third it within the effected capacity. Third it within the effected capacity. Third it need to keep the routing weights non- need to keep the routing weights non- need to keep the routing weights non- negative.

negative. negative. With this the controller control plane With this the controller control plane With this the controller control plane get the routing way from the optimizer get the routing way from the optimizer get the routing way from the optimizer and publish them and the data plane pull and publish them and the data plane pull and publish them and the data plane pull them and use them to make a globally them and use them to make a globally them and use them to make a globally optimized routing decision. optimized routing decision. optimized routing decision. And that's pretty much of my part and And that's pretty much of my part and And that's pretty much of my part and Lou will continue to talk about the Lou will continue to talk about the Lou will continue to talk about the protection mechanisms in the system. protection mechanisms in the system. protection mechanisms in the system. Thanks Chenu. So as AI engineers we all Thanks Chenu. So as AI engineers we all Thanks Chenu. So as AI engineers we all kind of know that production in many kind of know that production in many kind of know that production in many cases are not behaving in the most ideal cases are not behaving in the most ideal cases are not behaving in the most ideal case. So clusters can fail, GPUs or case. So clusters can fail, GPUs or case. So clusters can fail, GPUs or individual nodes can degrade degree and individual nodes can degrade degree and individual nodes can degrade degree and networking can just get to all kind of networking can just get to all kind of networking can just get to all kind of mysterious issues. So how do we keep our mysterious issues. So how do we keep our mysterious issues. So how do we keep our production system uh heresy as much as production system uh heresy as much as production system uh heresy as much as possible under the heavy load? The first possible under the heavy load? The first possible under the heavy load? The first thing we have is the penalties. thing we have is the penalties. thing we have is the penalties. Basically when an engine is an outlier, Basically when an engine is an outlier, Basically when an engine is an outlier, we detect the try to reduce the routing we detect the try to reduce the routing we detect the try to reduce the routing weight to that engine. In that way, we weight to that engine. In that way, we weight to that engine. In that way, we give it a chance to either recover by give it a chance to either recover by give it a chance to either recover by themselves if there's a uh if it's some themselves if there's a uh if it's some themselves if there's a uh if it's some transient issue or we can have a human transient issue or we can have a human transient issue or we can have a human intervented out or replace the faulty intervented out or replace the faulty intervented out or replace the faulty hardware. And secondly, the retries hardware. And secondly, the retries hardware. And secondly, the retries which is a very common technique used to which is a very common technique used to which is a very common technique used to mitigate a problems. However, during mitigate a problems. However, during mitigate a problems. However, during some cases, it's actually could make some cases, it's actually could make some cases, it's actually could make them even worse like when the system is

them even worse like when the system is them even worse like when the system is very close to like a tip over or very very close to like a tip over or very very close to like a tip over or very heavy utilized retries, we are send more heavy utilized retries, we are send more heavy utilized retries, we are send more load and this more load we are cause load and this more load we are cause load and this more load we are cause more failure and cause more retries more failure and cause more retries more failure and cause more retries which is infamous retry storm. So we which is infamous retry storm. So we which is infamous retry storm. So we incre implemented caps or budget to incre implemented caps or budget to incre implemented caps or budget to constant retries into a acceptable constant retries into a acceptable constant retries into a acceptable region and this is actually even need to region and this is actually even need to region and this is actually even need to be dynamic because in the happy time or be dynamic because in the happy time or be dynamic because in the happy time or in the normal time we can tolerate a lot in the normal time we can tolerate a lot in the normal time we can tolerate a lot more retries than when the system are more retries than when the system are more retries than when the system are heavily utilized. And finally we have heavily utilized. And finally we have heavily utilized. And finally we have the load shedding which is our last the load shedding which is our last the load shedding which is our last result when the production capac uh result when the production capac uh result when the production capac uh capacity couldn't meet the increasing capacity couldn't meet the increasing capacity couldn't meet the increasing amount of inference demands. So we amount of inference demands. So we amount of inference demands. So we instead try to have all the system fail. instead try to have all the system fail. instead try to have all the system fail. We basically proactively load shed a We basically proactively load shed a We basically proactively load shed a portion of the traffic to have the portion of the traffic to have the portion of the traffic to have the system degraded gracefully. So that system degraded gracefully. So that system degraded gracefully. So that pretty much concludes our talk today and pretty much concludes our talk today and pretty much concludes our talk today and uh thanks for joining us. Uh both of us uh thanks for joining us. Uh both of us uh thanks for joining us. Uh both of us will be around in our uh booth area this will be around in our uh booth area this will be around in our uh booth area this afternoon. So if you have further afternoon. So if you have further afternoon. So if you have further questions, feel free to walk uh to the questions, feel free to walk uh to the questions, feel free to walk uh to the area and and chat with us. Thank you.

area and and chat with us. Thank you. area and and chat with us. Thank you. [applause]