Productionizing LLM Gateways: Architecture, Tradeoffs and Hard Lessons — Kanish Manuja, Twilio

I'm Kanesh Manuja. I'm a principal I'm Kanesh Manuja. I'm a principal engineer at Twilio. engineer at Twilio. engineer at Twilio. Let's start with a quick show of hands. Let's start with a quick show of hands. Let's start with a quick show of hands. Who here has seen the message, something Who here has seen the message, something Who here has seen the message, something went wrong. Please try again. went wrong. Please try again. went wrong. Please try again. Well, we have a few lucky ones and a few Well, we have a few lucky ones and a few Well, we have a few lucky ones and a few that have had a good lunch. Um, so that have had a good lunch. Um, so that have had a good lunch. Um, so behind that simple message is actually a behind that simple message is actually a behind that simple message is actually a system that is very complex system that is very complex system that is very complex that serves you that message despite the that serves you that message despite the that serves you that message despite the model providers being down. model providers being down. model providers being down. And that's what we're going to And that's what we're going to And that's what we're going to productionize today or discuss productionize today or discuss productionize today or discuss productionizing today. productionizing today. productionizing today. So what is an LM gateway? An LLM gateway So what is an LM gateway? An LLM gateway So what is an LM gateway? An LLM gateway is an entry point or a middleware is an entry point or a middleware is an entry point or a middleware between your apps and the model between your apps and the model between your apps and the model providers behind them. It does a bunch providers behind them. It does a bunch providers behind them. It does a bunch of things. Routing, authentication, of things. Routing, authentication, of things. Routing, authentication, fallback, rate limits, all kinds of fallback, rate limits, all kinds of fallback, rate limits, all kinds of governance that you can think of. governance that you can think of. governance that you can think of. And right at the heart of the gateway is And right at the heart of the gateway is And right at the heart of the gateway is a fight between four things. It's a fight between four things. It's a fight between four things. It's availability, latency, your guardrails availability, latency, your guardrails availability, latency, your guardrails and costs.

and costs. and costs. In case of a degradation, you cannot In case of a degradation, you cannot In case of a degradation, you cannot maximize all four. You need to pick what maximize all four. You need to pick what maximize all four. You need to pick what you want. So with this talk, if you use you want. So with this talk, if you use you want. So with this talk, if you use an LLM gateway, I want you I want to an LLM gateway, I want you I want to an LLM gateway, I want you I want to help you to make that trade-off for your help you to make that trade-off for your help you to make that trade-off for your use case. And if you design a gateway, I use case. And if you design a gateway, I use case. And if you design a gateway, I want you to design or provide those want you to design or provide those want you to design or provide those levers to your callers and customers u levers to your callers and customers u levers to your callers and customers u so that your customers are happy. so that your customers are happy. so that your customers are happy. Let's start with availability. If you have a single model provider, If you have a single model provider, their ceiling is your ceiling. Their their ceiling is your ceiling. Their their ceiling is your ceiling. Their outage is your outage. So in typical software engineering, the So in typical software engineering, the way you tackle unreliable dependency is way you tackle unreliable dependency is way you tackle unreliable dependency is by retrying. by retrying. by retrying. Retrying with exponential backoffs, with Retrying with exponential backoffs, with Retrying with exponential backoffs, with jitters. And when all of that fails, you jitters. And when all of that fails, you jitters. And when all of that fails, you have a circuit breaker that trips after have a circuit breaker that trips after have a circuit breaker that trips after you've seen sufficient failures and you you've seen sufficient failures and you you've seen sufficient failures and you stop calling the damn thing.

stop calling the damn thing. stop calling the damn thing. This is not enough for LLMs. LLMs are This is not enough for LLMs. LLMs are This is not enough for LLMs. LLMs are very different compared to your fast very different compared to your fast very different compared to your fast cheap APIs that you retry on. Retrying cheap APIs that you retry on. Retrying cheap APIs that you retry on. Retrying an LLM API eats into your latency budget an LLM API eats into your latency budget an LLM API eats into your latency budget really fast. And also tripping over a really fast. And also tripping over a really fast. And also tripping over a circuit breaker when you have another circuit breaker when you have another circuit breaker when you have another perfectly fine model provider to route perfectly fine model provider to route perfectly fine model provider to route to doesn't make sense. You should use to doesn't make sense. You should use to doesn't make sense. You should use the second model provider. And third, as the second model provider. And third, as the second model provider. And third, as I said, the calls are slow and I said, the calls are slow and I said, the calls are slow and expensive. So blind retries just expensive. So blind retries just expensive. So blind retries just multiply your cost and your tail multiply your cost and your tail multiply your cost and your tail latencies. So what is a better idea here? It is So what is a better idea here? It is actually a per request fallback. What actually a per request fallback. What actually a per request fallback. What that means is you can actually try model that means is you can actually try model that means is you can actually try model provider A and then in sequence try provider A and then in sequence try provider A and then in sequence try model provider B if your request to model provider B if your request to model provider B if your request to model provider A fails. Another option model provider A fails. Another option model provider A fails. Another option to consider here is you can fire to consider here is you can fire to consider here is you can fire requests to both the providers in requests to both the providers in requests to both the providers in parallel. But that's only if you're parallel. But that's only if you're parallel. But that's only if you're highly highly obsessed with latencies highly highly obsessed with latencies highly highly obsessed with latencies because that's just going to double your because that's just going to double your because that's just going to double your cost.

cost. cost. Some of the similar circuit breaking Some of the similar circuit breaking Some of the similar circuit breaking patterns apply here to LMS as well. If patterns apply here to LMS as well. If patterns apply here to LMS as well. If you know that your primary has been you know that your primary has been you know that your primary has been failing for some time, it doesn't make failing for some time, it doesn't make failing for some time, it doesn't make sense to try it again. You put it, you sense to try it again. You put it, you sense to try it again. You put it, you take it out of the load balancer or your take it out of the load balancer or your take it out of the load balancer or your request path and put it in a cool down request path and put it in a cool down request path and put it in a cool down and then after a few minutes have and then after a few minutes have and then after a few minutes have passed, try putting that back again. passed, try putting that back again. passed, try putting that back again. One interesting choice that you have to One interesting choice that you have to One interesting choice that you have to make here is where your failure counts make here is where your failure counts make here is where your failure counts live. live. live. You can decide to have the failure You can decide to have the failure You can decide to have the failure counts live in memory on the instances counts live in memory on the instances counts live in memory on the instances that are serving your traffic or you can that are serving your traffic or you can that are serving your traffic or you can have shared infra where your failure have shared infra where your failure have shared infra where your failure counts are shared across the fleet. counts are shared across the fleet. counts are shared across the fleet. There are trade-offs. There are trade-offs. There are trade-offs. If you want quick failovers, then If you want quick failovers, then If you want quick failovers, then fleetwide helps. And with instance uh fleetwide helps. And with instance uh fleetwide helps. And with instance uh with local state counters the issue that with local state counters the issue that with local state counters the issue that you run into is whenever you change your you run into is whenever you change your you run into is whenever you change your deployment size your configuration and deployment size your configuration and deployment size your configuration and your expectations change. So something your expectations change. So something your expectations change. So something to consider.

to consider. to consider. What that clean diagram did not really What that clean diagram did not really What that clean diagram did not really show you are some of the other gotchas show you are some of the other gotchas show you are some of the other gotchas that I'm going to discuss. So fallbacks that I'm going to discuss. So fallbacks that I'm going to discuss. So fallbacks are not transparent. are not transparent. are not transparent. While the industry is converging on an While the industry is converging on an While the industry is converging on an OpenAI API compatible format, I would OpenAI API compatible format, I would OpenAI API compatible format, I would say there are still nuances. So you need say there are still nuances. So you need say there are still nuances. So you need to really test your fallbacks well. They to really test your fallbacks well. They to really test your fallbacks well. They can have differences in your tool can have differences in your tool can have differences in your tool calling schemas, token limits, stop calling schemas, token limits, stop calling schemas, token limits, stop reasons and what have you. So with LM reasons and what have you. So with LM reasons and what have you. So with LM gateways, you can have a normalization gateways, you can have a normalization gateways, you can have a normalization layer that can ensure that you can do layer that can ensure that you can do layer that can ensure that you can do cross provider fallbacks as well. cross provider fallbacks as well. cross provider fallbacks as well. Another thing is streaming it. it. So essentially nobody wants to wait for So essentially nobody wants to wait for So essentially nobody wants to wait for 30 seconds to have a wall of text appear 30 seconds to have a wall of text appear 30 seconds to have a wall of text appear in front of them. So there are use cases in front of them. So there are use cases in front of them. So there are use cases where streaming is absolutely required. where streaming is absolutely required. where streaming is absolutely required. But it comes as at a cost. You trade But it comes as at a cost. You trade But it comes as at a cost. You trade away your levers. You cannot once you away your levers. You cannot once you away your levers. You cannot once you have decided to go with provider A, you have decided to go with provider A, you have decided to go with provider A, you have to continue going with provider A.

have to continue going with provider A. have to continue going with provider A. You cannot mid-stream change the You cannot mid-stream change the You cannot mid-stream change the providers. Whatever has been sent to the providers. Whatever has been sent to the providers. Whatever has been sent to the client, it's done. And that's where the client, it's done. And that's where the client, it's done. And that's where the something uh went wrong message, that's something uh went wrong message, that's something uh went wrong message, that's the one that you see. It's not because the one that you see. It's not because the one that you see. It's not because of laziness. It's by design uh that you of laziness. It's by design uh that you of laziness. It's by design uh that you see that and it's one of the trade-offs. see that and it's one of the trade-offs. see that and it's one of the trade-offs. I would like to call out one other thing I would like to call out one other thing I would like to call out one other thing where I've seen teams trip over and over where I've seen teams trip over and over where I've seen teams trip over and over again. They really provision and test again. They really provision and test again. They really provision and test their primary providers really well, but their primary providers really well, but their primary providers really well, but they the second provider, the fallback they the second provider, the fallback they the second provider, the fallback provider doesn't necessarily get the provider doesn't necessarily get the provider doesn't necessarily get the same level of love. And I would argue same level of love. And I would argue same level of love. And I would argue that your throughputs or your capacity that your throughputs or your capacity that your throughputs or your capacity or your headroom should be even higher or your headroom should be even higher or your headroom should be even higher for the second provider or the fallback for the second provider or the fallback for the second provider or the fallback provider because that's your last line provider because that's your last line provider because that's your last line of defense. If that goes down, your of defense. If that goes down, your of defense. If that goes down, your application goes down. Let's discuss latencies. Let's discuss latencies. Availability failures are right in your Availability failures are right in your Availability failures are right in your face. They fail. You get alarmed. You face. They fail. You get alarmed. You face. They fail. You get alarmed. You get paged. But high latencies can be the get paged. But high latencies can be the get paged. But high latencies can be the quiet ones. And they need to receive quiet ones. And they need to receive quiet ones. And they need to receive more love um than I would say tuning more love um than I would say tuning more love um than I would say tuning your services for just availability.

One thing to call out, a gateway may run One thing to call out, a gateway may run mixed workloads mixed workloads mixed workloads and you can have embedding embedding and you can have embedding embedding and you can have embedding embedding requests that takes just less than a requests that takes just less than a requests that takes just less than a second. You can have classification second. You can have classification second. You can have classification requests that take less than a second. requests that take less than a second. requests that take less than a second. Uh you have chat requests taking 3 Uh you have chat requests taking 3 Uh you have chat requests taking 3 seconds and reasoning requests taking a seconds and reasoning requests taking a seconds and reasoning requests taking a long time. long time. long time. Quick show of hands. If you measure Quick show of hands. If you measure Quick show of hands. If you measure your aggregate latency for your entire your aggregate latency for your entire your aggregate latency for your entire service. service. service. Well, that was a trick question. Sorry. Well, that was a trick question. Sorry. Well, that was a trick question. Sorry. You shouldn't. It doesn't make sense. You shouldn't. It doesn't make sense. You shouldn't. It doesn't make sense. It's a lie. You should be tracking your It's a lie. You should be tracking your It's a lie. You should be tracking your P99 per model per route, not a gateway P99 per model per route, not a gateway P99 per model per route, not a gateway wide number. Gateway wide number doesn't wide number. Gateway wide number doesn't wide number. Gateway wide number doesn't make sense, especially if you're running make sense, especially if you're running make sense, especially if you're running mixed workloads. And I hope you're not u mixed workloads. And I hope you're not u mixed workloads. And I hope you're not u for those who raise your hand. Another for those who raise your hand. Another for those who raise your hand. Another thing that can really I cannot emphasize thing that can really I cannot emphasize thing that can really I cannot emphasize this enough is for you to set timeouts this enough is for you to set timeouts this enough is for you to set timeouts on per model class per route. on per model class per route. on per model class per route. That's where that's the number one root That's where that's the number one root That's where that's the number one root cause of your silent outage. If you cause of your silent outage. If you cause of your silent outage. If you don't have a timeout, your gateway don't have a timeout, your gateway don't have a timeout, your gateway thinks you're hap your request is being thinks you're hap your request is being thinks you're hap your request is being happily served while it is not. And I'll happily served while it is not. And I'll happily served while it is not. And I'll leave you with this message for for leave you with this message for for leave you with this message for for latencies. Um, specifically a reasoning latencies. Um, specifically a reasoning latencies. Um, specifically a reasoning models normal is actually a chat models models normal is actually a chat models models normal is actually a chat models outage. So you definitely need to track outage. So you definitely need to track outage. So you definitely need to track latency per route.

Okay, this is the most painful or this Okay, this is the most painful or this the slide that has given me the most the slide that has given me the most the slide that has given me the most scarse which is reasoning and router scarse which is reasoning and router scarse which is reasoning and router models. So this is where truly the models. So this is where truly the models. So this is where truly the latency is unpredictable latency is unpredictable latency is unpredictable and reasoning models they do not give and reasoning models they do not give and reasoning models they do not give you you you they they're highly undeterministic more they they're highly undeterministic more they they're highly undeterministic more deterministic undeterministic than your deterministic undeterministic than your deterministic undeterministic than your normal models. You cannot set the normal models. You cannot set the normal models. You cannot set the temperature to zero in many cases and temperature to zero in many cases and temperature to zero in many cases and the same prompt can take somewhere from the same prompt can take somewhere from the same prompt can take somewhere from 2 seconds to 60 seconds and we've seen 2 seconds to 60 seconds and we've seen 2 seconds to 60 seconds and we've seen that in production where P99 suddenly that in production where P99 suddenly that in production where P99 suddenly popped to 60 seconds for no good reason. popped to 60 seconds for no good reason. popped to 60 seconds for no good reason. So that's So that's So that's while there's no magical solution to it. while there's no magical solution to it. while there's no magical solution to it. I would recommend that you at least I would recommend that you at least I would recommend that you at least start with fixing the reasoning level start with fixing the reasoning level start with fixing the reasoning level per route. So with router models, they per route. So with router models, they per route. So with router models, they hide that abstraction behind you. Like hide that abstraction behind you. Like hide that abstraction behind you. Like they pick which models to run and I they pick which models to run and I they pick which models to run and I would highly recommend that you at least would highly recommend that you at least would highly recommend that you at least make as much uh you make requests as make as much uh you make requests as make as much uh you make requests as determinist deterministic as possible determinist deterministic as possible determinist deterministic as possible with an undeterministic system.

with an undeterministic system. with an undeterministic system. Another idea is hedging the tail. You Another idea is hedging the tail. You Another idea is hedging the tail. You can have a you can fire another request can have a you can fire another request can have a you can fire another request if your primary request actually if your primary request actually if your primary request actually consumed let's say P90 of your latency consumed let's say P90 of your latency consumed let's say P90 of your latency budget. budget. budget. This can hedge the t this can really This can hedge the t this can really This can hedge the t this can really hedge the P99 tail u for for your hedge the P99 tail u for for your hedge the P99 tail u for for your services. services. services. All right. This is one of my favorite All right. This is one of my favorite All right. This is one of my favorite ones. Um ones. Um ones. Um to keep your model secure you need to to keep your model secure you need to to keep your model secure you need to have guardrails. have guardrails. have guardrails. And with that, guardrails are necessary And with that, guardrails are necessary And with that, guardrails are necessary for preventing your services from prompt for preventing your services from prompt for preventing your services from prompt injection attacks, keeping PII filters injection attacks, keeping PII filters injection attacks, keeping PII filters in place, having toxicity filters, in place, having toxicity filters, in place, having toxicity filters, keeping the LMS to stop swearing at your keeping the LMS to stop swearing at your keeping the LMS to stop swearing at your customers, all those good things. But customers, all those good things. But customers, all those good things. But just like a model provider, there are just like a model provider, there are just like a model provider, there are trade-offs, too. Guardrails are just trade-offs, too. Guardrails are just trade-offs, too. Guardrails are just like another service that can go down like another service that can go down like another service that can go down that can be unreliable and that's where that can be unreliable and that's where that can be unreliable and that's where you need to choose do you fail open or you need to choose do you fail open or you need to choose do you fail open or do you fail close when I say fail open do you fail close when I say fail open do you fail close when I say fail open you can still serve the request even if you can still serve the request even if you can still serve the request even if your guardrails are down fail close you your guardrails are down fail close you your guardrails are down fail close you block the request and say hey I'm not block the request and say hey I'm not block the request and say hey I'm not available that's the trade-off between available that's the trade-off between available that's the trade-off between availability and security to certain availability and security to certain availability and security to certain extent while there's a no universal extent while there's a no universal extent while there's a no universal answer it really depends on your use answer it really depends on your use answer it really depends on your use case you can decide like for example a case you can decide like for example a case you can decide like for example a toxicity filter if it's not up and toxicity filter if it's not up and toxicity filter if it's not up and running you can still serve that running you can still serve that running you can still serve that request.

request. request. So the default choice should be the So the default choice should be the So the default choice should be the worst case that you can live with. There are a few things that you can There are a few things that you can actually do to improve the behavior of actually do to improve the behavior of actually do to improve the behavior of your systems in face of uh you know your systems in face of uh you know your systems in face of uh you know guardrails being down and and managing guardrails being down and and managing guardrails being down and and managing just unreliability of the guardrails just unreliability of the guardrails just unreliability of the guardrails themselves. So the first is time budget. themselves. So the first is time budget. themselves. So the first is time budget. Your request should never be bound by Your request should never be bound by Your request should never be bound by your guardrail timing. It should always your guardrail timing. It should always your guardrail timing. It should always be the LM that is the rate determining be the LM that is the rate determining be the LM that is the rate determining step. So make sure that you have step. So make sure that you have step. So make sure that you have timeouts in place and those guardrails timeouts in place and those guardrails timeouts in place and those guardrails run with a specific time budget. run with a specific time budget. run with a specific time budget. Another important thing is fallback. Another important thing is fallback. Another important thing is fallback. You've heard, you probably know and I've You've heard, you probably know and I've You've heard, you probably know and I've talked about it. We always discuss talked about it. We always discuss talked about it. We always discuss fallbacks with regards to model fallbacks with regards to model fallbacks with regards to model providers, but guardrails are critical providers, but guardrails are critical providers, but guardrails are critical services too where you can consider services too where you can consider services too where you can consider fallbacks, have secondary provider, fallbacks, have secondary provider, fallbacks, have secondary provider, secondary checks, cache decisions uh to secondary checks, cache decisions uh to secondary checks, cache decisions uh to keep your service available when a keep your service available when a keep your service available when a guardrail provider is down.

guardrail provider is down. guardrail provider is down. Another interesting choice that pops up Another interesting choice that pops up Another interesting choice that pops up with regards to guardrails is the with regards to guardrails is the with regards to guardrails is the placement of the guardrails. placement of the guardrails. placement of the guardrails. Typically, you can place the guardrail Typically, you can place the guardrail Typically, you can place the guardrail in three ways. You can have a pre- hook in three ways. You can have a pre- hook in three ways. You can have a pre- hook that runs where the guardrail actually that runs where the guardrail actually that runs where the guardrail actually runs on the input. You can and that's runs on the input. You can and that's runs on the input. You can and that's probably the safest uh but it does add probably the safest uh but it does add probably the safest uh but it does add serial latency uh to your requests. serial latency uh to your requests. serial latency uh to your requests. Another one is in parallel. This is one Another one is in parallel. This is one Another one is in parallel. This is one of my favorites, but just to call out, of my favorites, but just to call out, of my favorites, but just to call out, streaming wouldn't work well here with streaming wouldn't work well here with streaming wouldn't work well here with with parallel. So if you're specially with parallel. So if you're specially with parallel. So if you're specially producing structured output, please producing structured output, please producing structured output, please don't stream them. Uh try to save your don't stream them. Uh try to save your don't stream them. Uh try to save your latencies and run run these guardrails latencies and run run these guardrails latencies and run run these guardrails concurrently for your structured concurrently for your structured concurrently for your structured outputs. Another one is post hooks. The outputs. Another one is post hooks. The outputs. Another one is post hooks. The these are best for um output monitoring, these are best for um output monitoring, these are best for um output monitoring, auditing your outputs and and so forth. So, so far we've all I've discussed all So, so far we've all I've discussed all the things that can go wrong with the things that can go wrong with the things that can go wrong with regards to our dependencies.

regards to our dependencies. regards to our dependencies. We haven't discussed that we are We haven't discussed that we are We haven't discussed that we are actually adding another dependency in actually adding another dependency in actually adding another dependency in the request path itself which is the the request path itself which is the the request path itself which is the central or which is the LM gateway central or which is the LM gateway central or which is the LM gateway itself. There are a few things where we itself. There are a few things where we itself. There are a few things where we have been bitten by u and we've learned have been bitten by u and we've learned have been bitten by u and we've learned some lessons that I want to share with some lessons that I want to share with some lessons that I want to share with you. If you're working on an LLM gateway you. If you're working on an LLM gateway you. If you're working on an LLM gateway or using one, one is shared limits. or using one, one is shared limits. or using one, one is shared limits. Make sure that your API keys are Make sure that your API keys are Make sure that your API keys are segregated per route, per use case to segregated per route, per use case to segregated per route, per use case to the most granular possible uh to the the most granular possible uh to the the most granular possible uh to the most granular thing that you can most granular thing that you can most granular thing that you can imagine. U imagine. U imagine. U having a noisy tenant can be one of the having a noisy tenant can be one of the having a noisy tenant can be one of the biggest problems here. biggest problems here. biggest problems here. Another thing is load shedding. This is Another thing is load shedding. This is Another thing is load shedding. This is a feature that you should uh as part of a feature that you should uh as part of a feature that you should uh as part of your runbooks, game days, uh make sure your runbooks, game days, uh make sure your runbooks, game days, uh make sure that the gateway that you're using that the gateway that you're using that the gateway that you're using supports load shedding because when you supports load shedding because when you supports load shedding because when you have a retry storm, it becomes really have a retry storm, it becomes really have a retry storm, it becomes really hard to just scale out. You cannot hard to just scale out. You cannot hard to just scale out. You cannot simply scale out services that is under simply scale out services that is under simply scale out services that is under a retry storm and all these web servers a retry storm and all these web servers a retry storm and all these web servers they have an internal queue and they're they have an internal queue and they're they have an internal queue and they're configurable. Make sure that they're configurable. Make sure that they're configurable. Make sure that they're bounded and they cannot request they bounded and they cannot request they bounded and they cannot request they cannot accept requests that are cannot accept requests that are cannot accept requests that are unbounded. And if you want to have some unbounded. And if you want to have some unbounded. And if you want to have some custom logic, you can even have traffic custom logic, you can even have traffic custom logic, you can even have traffic prioritization here as well to make sure prioritization here as well to make sure prioritization here as well to make sure under load your most important use cases under load your most important use cases under load your most important use cases get served. Well, last thing that I wanted to discuss is last thing that I wanted to discuss is the whole idea of a central gateway

the whole idea of a central gateway the whole idea of a central gateway itself. It is a single point of failure. itself. It is a single point of failure. itself. It is a single point of failure. So if you're thinking of having a So if you're thinking of having a So if you're thinking of having a central gateway for your entire company central gateway for your entire company central gateway for your entire company for to LLMs, I would recommend rethink for to LLMs, I would recommend rethink for to LLMs, I would recommend rethink that and see what are the reasons that that and see what are the reasons that that and see what are the reasons that you want it. What I've noticed is that you want it. What I've noticed is that you want it. What I've noticed is that in most scenarios, it's not the central in most scenarios, it's not the central in most scenarios, it's not the central gateway that they want. They want gateway that they want. They want gateway that they want. They want centralized governance. centralized governance. centralized governance. And there is a path forward where you And there is a path forward where you And there is a path forward where you can actually decentralize the gateway can actually decentralize the gateway can actually decentralize the gateway and still centralize government and still centralize government and still centralize government governance. So do not try to centralize governance. So do not try to centralize governance. So do not try to centralize your traffic but you can have plugins, your traffic but you can have plugins, your traffic but you can have plugins, you can have custom code that can you can have custom code that can you can have custom code that can centralize your governance. Uh centralize your governance. Uh centralize your governance. Uh governance can be in the form of cost governance can be in the form of cost governance can be in the form of cost tracking, rate limit managing management tracking, rate limit managing management tracking, rate limit managing management and there are other solutions possible. and there are other solutions possible. and there are other solutions possible. So explore those before you chart on So explore those before you chart on So explore those before you chart on having one central gateway for your having one central gateway for your having one central gateway for your entire company. It can be managed by a entire company. It can be managed by a entire company. It can be managed by a single team, but I wouldn't recommend single team, but I wouldn't recommend single team, but I wouldn't recommend deploying it as a single deployment for deploying it as a single deployment for deploying it as a single deployment for the entire company even though it's the entire company even though it's the entire company even though it's distributed.

distributed. distributed. With that said, I want to end this talk With that said, I want to end this talk With that said, I want to end this talk on a personal note. So, it is my son's on a personal note. So, it is my son's on a personal note. So, it is my son's birthday today and I'm here talking to birthday today and I'm here talking to birthday today and I'm here talking to strangers about circuit breaking. So the strangers about circuit breaking. So the strangers about circuit breaking. So the least you can do for me is please go and least you can do for me is please go and least you can do for me is please go and prevent one incident for me and for your prevent one incident for me and for your prevent one incident for me and for your customers. Thank you. If you have any customers. Thank you. If you have any customers. Thank you. If you have any questions. Yeah.