Serving 2 Million Models Without Melting: Scaling the Hugging Face Hub — Arek Borucki, Hugging Face

Good afternoon everyone. Good afternoon everyone. I have a question. I have a question. I have a question. How many of you How many of you How many of you knows hugging face? knows hugging face? knows hugging face? Nice. Nice. Nice. How many of you already use hugging How many of you already use hugging How many of you already use hugging face? face? face? Amazing. Almost everyone. Amazing. Almost everyone. Amazing. Almost everyone. But I think we still have opportunity to But I think we still have opportunity to But I think we still have opportunity to grow our usage. My name is Ar Borutki. I grow our usage. My name is Ar Borutki. I grow our usage. My name is Ar Borutki. I work as machine learning platform and work as machine learning platform and work as machine learning platform and database engineer at hugging face. database engineer at hugging face. database engineer at hugging face. Today Today Today I would like to walk you through how I would like to walk you through how I would like to walk you through how hugging faced scaled hugging faced scaled hugging faced scaled infrastructure infrastructure infrastructure and how we ended up serving 3 million and how we ended up serving 3 million and how we ended up serving 3 million models models models to developers around the world. to developers around the world. to developers around the world. I would like to share architectural I would like to share architectural I would like to share architectural decisions we made, challenges we faced, decisions we made, challenges we faced, decisions we made, challenges we faced, and lessons we learned while scaling one and lessons we learned while scaling one and lessons we learned while scaling one of the fastest growing open-source AI of the fastest growing open-source AI of the fastest growing open-source AI communities in the world.

communities in the world. communities in the world. I hope you will enjoy it and let's get I hope you will enjoy it and let's get I hope you will enjoy it and let's get started. Before I dive into technical details, Before I dive into technical details, let's talk about scale. let's talk about scale. let's talk about scale. Today, Hugging Face serves more than 14 Today, Hugging Face serves more than 14 Today, Hugging Face serves more than 14 million users and this number is growing million users and this number is growing million users and this number is growing very fast, especially in the last couple very fast, especially in the last couple very fast, especially in the last couple of months. of months. of months. We host three million public models, We host three million public models, We host three million public models, 1 million data sets, 1 million data sets, 1 million data sets, 50,000 organizations, 50,000 organizations, 50,000 organizations, and not only hobbyists or scientists. and not only hobbyists or scientists. and not only hobbyists or scientists. More than 30% More than 30% More than 30% of Fortune 500 use hugging face as a of Fortune 500 use hugging face as a of Fortune 500 use hugging face as a part of AI workflows. part of AI workflows. part of AI workflows. Just to give you some perspective, Just to give you some perspective, Just to give you some perspective, few years ago we had 20,000 models.

few years ago we had 20,000 models. few years ago we had 20,000 models. Today 3 million. Today 3 million. Today 3 million. It is around It is around It is around 150x 150x 150x increase in just last couple of years. increase in just last couple of years. increase in just last couple of years. And this grow is exactly why I'm here And this grow is exactly why I'm here And this grow is exactly why I'm here today talking about infrastructural today talking about infrastructural today talking about infrastructural decisions that keep the hub healthy at decisions that keep the hub healthy at decisions that keep the hub healthy at scale. This is how fast the number of public This is how fast the number of public models is growing on the hub. Every big models is growing on the hub. Every big models is growing on the hub. Every big release like llama or deepseek generated release like llama or deepseek generated release like llama or deepseek generated thousands of new models on top thousands of new models on top thousands of new models on top and our infrastructure needs to handle and our infrastructure needs to handle and our infrastructure needs to handle that that that and it is not only models and it is not only models and it is not only models also data sets. also data sets. also data sets. In 2022, we had 10K. In 2022, we had 10K. In 2022, we had 10K. In 2024, In 2024, In 2024, 100K.

100K. 100K. Less than a year ago, we had 500K. Less than a year ago, we had 500K. Less than a year ago, we had 500K. Today, 1 million. Today, 1 million. Today, 1 million. All this data All this data All this data must be stored, must be stored, must be stored, indexed, and also must be searchable. indexed, and also must be searchable. indexed, and also must be searchable. And that's the hardest part. And this is also the reason why we had And this is also the reason why we had to rethink our search. to rethink our search. to rethink our search. At 20,000 models, any query is fast. At 20,000 models, any query is fast. At 20,000 models, any query is fast. Even without an index, trust me, no one Even without an index, trust me, no one Even without an index, trust me, no one would notice. At 3 million, same would notice. At 3 million, same would notice. At 3 million, same approach breaks. approach breaks. approach breaks. Imagine what would you do if the hub Imagine what would you do if the hub Imagine what would you do if the hub search would be slow. search would be slow. search would be slow. you would just leave and go somewhere you would just leave and go somewhere you would just leave and go somewhere else. And this is also what user are else. And this is also what user are else. And this is also what user are doing. They expect fast instant results. doing. They expect fast instant results. doing. They expect fast instant results. With 14 million users, With 14 million users, With 14 million users, even 1% even 1% even 1% is a not small number. It is 140,000 is a not small number. It is 140,000 is a not small number. It is 140,000 of people hitting slow search at scale.

at scale. P99 is much more important than P50 and P99 is much more important than P50 and P99 is much more important than P50 and we are paying a lots of attention to P99 we are paying a lots of attention to P99 we are paying a lots of attention to P99 and that's the reason why we invest and that's the reason why we invest and that's the reason why we invest in premputee tokens in premputee tokens in premputee tokens denormalize optimize for read collection denormalize optimize for read collection denormalize optimize for read collection in MongoDB in MongoDB in MongoDB full text search based on Apache lucine full text search based on Apache lucine full text search based on Apache lucine Kubernetes autoscaling Kubernetes autoscaling Kubernetes autoscaling and soon in database sharding. and soon in database sharding. and soon in database sharding. The next slides will show you how high level architecture high level architecture when user interact with the hugging face when user interact with the hugging face when user interact with the hugging face hub his request flows from the front end hub his request flows from the front end hub his request flows from the front end to the hub API. to the hub API. to the hub API. The hub is running on Kubernetes. The hub is running on Kubernetes. The hub is running on Kubernetes. Currently we are using horizontal pot Currently we are using horizontal pot Currently we are using horizontal pot autoscaler.

autoscaler. autoscaler. During spikes new ports scale up During spikes new ports scale up During spikes new ports scale up automatically to handle the load and automatically to handle the load and automatically to handle the load and scale back down when traffic drops. scale back down when traffic drops. scale back down when traffic drops. This help us to keep the hub healthy This help us to keep the hub healthy This help us to keep the hub healthy without manual intervention. without manual intervention. without manual intervention. Next the request Next the request Next the request goes to MongoDB Atlas goes to MongoDB Atlas goes to MongoDB Atlas which is source of true for our which is source of true for our which is source of true for our metadata. metadata. metadata. And there is one point that sometimes And there is one point that sometimes And there is one point that sometimes surprise people. surprise people. surprise people. MongoDB does not store the models MongoDB does not store the models MongoDB does not store the models themselves. themselves. themselves. It stores everything about the models. It stores everything about the models. It stores everything about the models. What doesn't mean in practice What doesn't mean in practice What doesn't mean in practice in MongoDB in MongoDB in MongoDB we hold all the metadata we hold all the metadata we hold all the metadata users users users repositories repositories repositories models models models data sets buckets data sets buckets data sets buckets spaces information spaces information spaces information configuration data billing data access configuration data billing data access configuration data billing data access control and more.

control and more. control and more. The actual models artifacts, The actual models artifacts, The actual models artifacts, tokenizer files, cart assets and tokenizer files, cart assets and tokenizer files, cart assets and configuration files are stored configuration files are stored configuration files are stored separately in cloud object storage such separately in cloud object storage such separately in cloud object storage such as AWSS3. as AWSS3. as AWSS3. This separation of concern This separation of concern This separation of concern let us scale metadata let us scale metadata let us scale metadata independently from binary storage and independently from binary storage and independently from binary storage and compute independently from both. compute independently from both. compute independently from both. We can optimize each component We can optimize each component We can optimize each component individually individually individually for specific workload. Now let's check how search works in Now let's check how search works in details. details. details. For example, someone wants to search the For example, someone wants to search the For example, someone wants to search the model on the hub and let's say that's model on the hub and let's say that's model on the hub and let's say that's llama. So someone type llama into llama. So someone type llama into llama. So someone type llama into hugging face search bar. His request hugging face search bar. His request hugging face search bar. His request flows through the hub to an optimized flows through the hub to an optimized flows through the hub to an optimized read collection on MongoDB. And this is read collection on MongoDB. And this is read collection on MongoDB. And this is not our main repo collection when we not our main repo collection when we not our main repo collection when we keep all the data. It's a separate keep all the data. It's a separate keep all the data. It's a separate denormalized copy denormalized copy denormalized copy only for reads and listings.

only for reads and listings. only for reads and listings. The key information is on the left. The key information is on the left. The key information is on the left. We tokenize model names on insert time, We tokenize model names on insert time, We tokenize model names on insert time, not at query time. not at query time. not at query time. For example, someone wants to publish For example, someone wants to publish For example, someone wants to publish model meta minus llama/ llama 3.18b. We split the long model name into small We split the long model name into small tokens like meta lama 3.18b and we store tokens like meta lama 3.18b and we store tokens like meta lama 3.18b and we store them in an array in MongoDB document. them in an array in MongoDB document. them in an array in MongoDB document. Next, Atlas search which is using Apache Next, Atlas search which is using Apache Next, Atlas search which is using Apache Lucin under the hood use autocomplete Lucin under the hood use autocomplete Lucin under the hood use autocomplete to find matching models instantly. This is example of single document from This is example of single document from our model collection. our model collection. our model collection. In this example, I'm using find one In this example, I'm using find one In this example, I'm using find one method. I want to find model ID meta method. I want to find model ID meta method. I want to find model ID meta minus llama/lama 3.1. So that's the minus llama/lama 3.1. So that's the minus llama/lama 3.1. So that's the model from the previous slide.

model from the previous slide. model from the previous slide. And I'm projecting only search token And I'm projecting only search token And I'm projecting only search token array. array. array. And we see that all those premputee And we see that all those premputee And we see that all those premputee tokens are part of this array. So we tokens are part of this array. So we tokens are part of this array. So we have metal lama 3.1 metal lama etc. have metal lama 3.1 metal lama etc. have metal lama 3.1 metal lama etc. Next Next Next there must be a query. there must be a query. there must be a query. In the past we were using classical In the past we were using classical In the past we were using classical MongoDB find method on models collection MongoDB find method on models collection MongoDB find method on models collection with reax operator with reax operator with reax operator and this reax operator we're searching and this reax operator we're searching and this reax operator we're searching in search tokens arrays models which are in search tokens arrays models which are in search tokens arrays models which are equal to llama and then we were s sorting results by and then we were s sorting results by trending score which is calculated every trending score which is calculated every trending score which is calculated every five minutes. This is number of five minutes. This is number of five minutes. This is number of downloads and number of likes downloads and number of likes downloads and number of likes is as far as I remember from the last is as far as I remember from the last is as far as I remember from the last seven days.

seven days. seven days. This solution was working well as long This solution was working well as long This solution was working well as long as data set was small. as data set was small. as data set was small. Reax doesn't scale well. So when our Reax doesn't scale well. So when our Reax doesn't scale well. So when our data set started to grow very quickly, data set started to grow very quickly, data set started to grow very quickly, we started to having problems with we started to having problems with we started to having problems with latency. latency. latency. So we decided to switch to Atlas search So we decided to switch to Atlas search So we decided to switch to Atlas search that's a feature which is using Apache that's a feature which is using Apache that's a feature which is using Apache lucine under the hood. lucine under the hood. lucine under the hood. So MongoDB doesn't provide in core So MongoDB doesn't provide in core So MongoDB doesn't provide in core MongoDB server full text search there is MongoDB server full text search there is MongoDB server full text search there is additional process MongoD additional process MongoD additional process MongoD uh this MongoDBT process is a wrapper uh this MongoDBT process is a wrapper uh this MongoDBT process is a wrapper around Apache lucine around Apache lucine around Apache lucine for end user users this is transparent for end user users this is transparent for end user users this is transparent you are just using Unifi MongoDB query you are just using Unifi MongoDB query you are just using Unifi MongoDB query API API API and and and when You when You when You use aggregation pip pipeline together use aggregation pip pipeline together use aggregation pip pipeline together with dollar search operator. MongoDB with dollar search operator. MongoDB with dollar search operator. MongoDB will know that you would like to search will know that you would like to search will know that you would like to search Apache Lucin index.

Apache Lucin index. Apache Lucin index. Obviously you need to put the name of Obviously you need to put the name of Obviously you need to put the name of this index which is in this scenario this index which is in this scenario this index which is in this scenario model search model search model search autocomplete model equal to llama path search tokens model equal to llama path search tokens and we still sort results by trending and we still sort results by trending and we still sort results by trending score and this solution is much more score and this solution is much more score and this solution is much more efficient efficient efficient and is so far scale Well, so we don't and is so far scale Well, so we don't and is so far scale Well, so we don't have any more latency issues in our have any more latency issues in our have any more latency issues in our search bar. First two results returned by previous First two results returned by previous query. First metal lama has trending query. First metal lama has trending query. First metal lama has trending score 33. score 33. score 33. Second one 14. But hugging face hub is not only search. But hugging face hub is not only search. We have hundreds of different services We have hundreds of different services We have hundreds of different services in hugging face which are utilizing in hugging face which are utilizing in hugging face which are utilizing which are using MongoDB which are using MongoDB which are using MongoDB to handle million of queries. We use to handle million of queries. We use to handle million of queries. We use seven nodes MongoDB clusters cluster seven nodes MongoDB clusters cluster seven nodes MongoDB clusters cluster with multiple machines. We can with multiple machines. We can with multiple machines. We can distribute queries across multiple distribute queries across multiple distribute queries across multiple nodes. So no single node become read nodes. So no single node become read nodes. So no single node become read bottleneck.

bottleneck. bottleneck. This is how it works. Application talk This is how it works. Application talk This is how it works. Application talk to the MongoDB cluster. All inserts, to the MongoDB cluster. All inserts, to the MongoDB cluster. All inserts, deletions or updates goes to single deletions or updates goes to single deletions or updates goes to single primary because only primary can handle primary because only primary can handle primary because only primary can handle them. them. them. However, However, However, we are distributing reads across we are distributing reads across we are distributing reads across multiple machines. multiple machines. multiple machines. We also have one analytic We also have one analytic We also have one analytic hidden node. What does it mean? This hidden node. What does it mean? This hidden node. What does it mean? This mean this node is invisible from mean this node is invisible from mean this node is invisible from application. MongoDB driver is not application. MongoDB driver is not application. MongoDB driver is not routing any queries to this hidden node. routing any queries to this hidden node. routing any queries to this hidden node. This node is still replicating data from This node is still replicating data from This node is still replicating data from primary primary primary but is not interacting with interacting but is not interacting with interacting but is not interacting with interacting with production traffic. with production traffic. with production traffic. We are connecting directly to this node We are connecting directly to this node We are connecting directly to this node and we use him for any kind of and we use him for any kind of and we use him for any kind of reporting reporting reporting traffic on or any kind of really heavy traffic on or any kind of really heavy traffic on or any kind of really heavy queries.

queries. queries. All secondaries continuously All secondaries continuously All secondaries continuously tail the oplog from primary keeping tail the oplog from primary keeping tail the oplog from primary keeping cluster in sync. cluster in sync. cluster in sync. Now let's have a look what actually is Now let's have a look what actually is Now let's have a look what actually is running on secondaries. running on secondaries. running on secondaries. First all queries which doesn't require First all queries which doesn't require First all queries which doesn't require the latest data goes to secondaries. the latest data goes to secondaries. the latest data goes to secondaries. Only queries that must Only queries that must Only queries that must have strong consistency have strong consistency have strong consistency stay on primary and we are paying a lots stay on primary and we are paying a lots stay on primary and we are paying a lots of attention to this. We are paying lots of attention to this. We are paying lots of attention to this. We are paying lots of attention to the queries which must of attention to the queries which must of attention to the queries which must run on primary. run on primary. run on primary. Second, complex aggregations. Second, complex aggregations. Second, complex aggregations. Aggregations pipelines that scan large Aggregations pipelines that scan large Aggregations pipelines that scan large amount of data, sort, group or transform amount of data, sort, group or transform amount of data, sort, group or transform the data the data the data should not go on primary. They are should not go on primary. They are should not go on primary. They are heavy. Secondaries are better placed for heavy. Secondaries are better placed for heavy. Secondaries are better placed for them.

them. them. Third, change streams. Third, change streams. Third, change streams. We react to data changes in real time We react to data changes in real time We react to data changes in real time for several reasons. For example, for several reasons. For example, for several reasons. For example, caching validation, sync to different caching validation, sync to different caching validation, sync to different data store technologies like for example data store technologies like for example data store technologies like for example AWS, Red Shift or for eventdriven AWS, Red Shift or for eventdriven AWS, Red Shift or for eventdriven workloads. workloads. workloads. Those kind of operations are also not Those kind of operations are also not Those kind of operations are also not very light and they should stay very light and they should stay very light and they should stay impossible on secondaries. impossible on secondaries. impossible on secondaries. Fourth, Fourth, Fourth, all ad hoc queries, reporting queries, all ad hoc queries, reporting queries, all ad hoc queries, reporting queries, maybe some experimental queries go to maybe some experimental queries go to maybe some experimental queries go to hidden MongoDB replicas set member which hidden MongoDB replicas set member which hidden MongoDB replicas set member which is isolated from production traffic. is isolated from production traffic. is isolated from production traffic. The pattern is simple. Primary should The pattern is simple. Primary should The pattern is simple. Primary should focus on what only primary can do. focus on what only primary can do. focus on what only primary can do. Anything else can be pushed to different Anything else can be pushed to different Anything else can be pushed to different machines.

However, with 14 million users, 3 However, with 14 million users, 3 million models and our grow soon single million models and our grow soon single million models and our grow soon single MongoDB replica set will not be enough. MongoDB replica set will not be enough. MongoDB replica set will not be enough. The next step is sharding. The next step is sharding. The next step is sharding. Sharding means scaling your database Sharding means scaling your database Sharding means scaling your database horizontally. horizontally. horizontally. Instead of putting full data set on one Instead of putting full data set on one Instead of putting full data set on one replic on one replica set cluster, we replic on one replica set cluster, we replic on one replica set cluster, we are going to cut data into pieces are going to cut data into pieces are going to cut data into pieces and put each piece on separate chart. and put each piece on separate chart. and put each piece on separate chart. Each shard will have his own replication Each shard will have his own replication Each shard will have his own replication primary and secondary. So we will keep primary and secondary. So we will keep primary and secondary. So we will keep replication just multiplied. replication just multiplied. replication just multiplied. The key difference between replica set The key difference between replica set The key difference between replica set cluster and sharded cluster is replica cluster and sharded cluster is replica cluster and sharded cluster is replica set keep full data set on each node. set keep full data set on each node. set keep full data set on each node. Sharded cluster keep only part of the Sharded cluster keep only part of the Sharded cluster keep only part of the data on each shard. And then if you want data on each shard. And then if you want data on each shard. And then if you want to scale horizontally more you are just to scale horizontally more you are just to scale horizontally more you are just adding more shards and then MongoDB adding more shards and then MongoDB adding more shards and then MongoDB balancer will balance data across all balancer will balance data across all balancer will balance data across all those shards. There is also shard key those shards. There is also shard key those shards. There is also shard key which must be selected. This is not which must be selected. This is not which must be selected. This is not trivial operation but this talk is not trivial operation but this talk is not trivial operation but this talk is not about choosing short key.

about choosing short key. about choosing short key. This way we are going to scale This way we are going to scale This way we are going to scale everything everything everything CPU CPU CPU memory memory memory storage storage storage reads and writes. Now let's have a look what is going on Now let's have a look what is going on the hub level. the hub level. the hub level. DHA is running on Kubernetes. DHA is running on Kubernetes. DHA is running on Kubernetes. Currently we are using horizontal pot Currently we are using horizontal pot Currently we are using horizontal pot autoscaler. When CPU or memory threshold autoscaler. When CPU or memory threshold autoscaler. When CPU or memory threshold go above target, Kubernetes adds new pot go above target, Kubernetes adds new pot go above target, Kubernetes adds new pot automatically to handle this spike and automatically to handle this spike and automatically to handle this spike and scale them back down when traffic drops. scale them back down when traffic drops. scale them back down when traffic drops. Our deployment hub deployment can scale Our deployment hub deployment can scale Our deployment hub deployment can scale from 10 to 500 bots depends of on the from 10 to 500 bots depends of on the from 10 to 500 bots depends of on the traffic. This is how we keep the hub traffic. This is how we keep the hub traffic. This is how we keep the hub healthy without manual interventions healthy without manual interventions healthy without manual interventions and without infrastructure and without infrastructure and without infrastructure overprovisioning. overprovisioning. overprovisioning. So this is also cost effective solution.

So this is also cost effective solution. So this is also cost effective solution. However, However, However, what happens if horizontal pot what happens if horizontal pot what happens if horizontal pot autoscaler autoscaler autoscaler want to add new ports but kubernetes want to add new ports but kubernetes want to add new ports but kubernetes does not have free nodes anymore. does not have free nodes anymore. does not have free nodes anymore. This is where second layer comes in. We This is where second layer comes in. We This is where second layer comes in. We are using cast AI for Kubernetes note are using cast AI for Kubernetes note are using cast AI for Kubernetes note autoscaling autoscaling autoscaling when pods are pending because there is when pods are pending because there is when pods are pending because there is no capacity and Kubernetes scheduler is no capacity and Kubernetes scheduler is no capacity and Kubernetes scheduler is not able to schedule them. Cast AAI is not able to schedule them. Cast AAI is not able to schedule them. Cast AAI is adding new nodes adding new nodes adding new nodes and then scheduler is able to schedule and then scheduler is able to schedule and then scheduler is able to schedule those spots. So we have two layers of those spots. So we have two layers of those spots. So we have two layers of scaling. First one is at deployment scaling. First one is at deployment scaling. First one is at deployment level. Second one is at infrastructure level. Second one is at infrastructure level. Second one is at infrastructure level via castai. level via castai. level via castai. But we are going to migrate horizontal But we are going to migrate horizontal But we are going to migrate horizontal port autoscaler to keta kubernetes port autoscaler to keta kubernetes port autoscaler to keta kubernetes eventdriven autoscaling. The difference eventdriven autoscaling. The difference eventdriven autoscaling. The difference HPA scale only based on CPU and memory.

HPA scale only based on CPU and memory. HPA scale only based on CPU and memory. KDA scale on real application metrics KDA scale on real application metrics KDA scale on real application metrics like request per second or like request per second or like request per second or event loop utilization. It means scaling event loop utilization. It means scaling event loop utilization. It means scaling is driven by actual workload is driven by actual workload is driven by actual workload not by resource utilization only. For not by resource utilization only. For not by resource utilization only. For example, pot can have low CPU but high example, pot can have low CPU but high example, pot can have low CPU but high request Q. KDA can see it. HPA not the best part of this architecture. the best part of this architecture. You never have to think about it. When You never have to think about it. When You never have to think about it. When you pop when you push the model, search you pop when you push the model, search you pop when you push the model, search the hub or download the model, it just the hub or download the model, it just the hub or download the model, it just works. This is what scaling medium works. This is what scaling medium works. This is what scaling medium models is really about. Keeping the user models is really about. Keeping the user models is really about. Keeping the user experience simple no matter how complex experience simple no matter how complex experience simple no matter how complex it gets under the hood. it gets under the hood. it gets under the hood. Thank you very much. It was pleasure for Thank you very much. It was pleasure for Thank you very much. It was pleasure for me to be here today and I wish nice day me to be here today and I wish nice day me to be here today and I wish nice day for all of you.