How Web Data Infrastructure Powers the Next Generation of AI — Patricija Žemaitytė, Oxylabs

>> Okay, hello everyone. >> Okay, hello everyone. So, mostly I talk today starts with So, mostly I talk today starts with So, mostly I talk today starts with models. This one starts somewhere less models. This one starts somewhere less models. This one starts somewhere less glamorous with infrastructure that glamorous with infrastructure that glamorous with infrastructure that decides whether those models get fresh, decides whether those models get fresh, decides whether those models get fresh, usable, real-time data at all. usable, real-time data at all. usable, real-time data at all. So, I work at Oxylabs and Oxylabs was So, I work at Oxylabs and Oxylabs was So, I work at Oxylabs and Oxylabs was established in 2015 and describes itself established in 2015 and describes itself established in 2015 and describes itself as a web intelligence platform and a as a web intelligence platform and a as a web intelligence platform and a premium proxy provider. premium proxy provider. premium proxy provider. In simple terms, we built infrastructure In simple terms, we built infrastructure In simple terms, we built infrastructure that allows companies to extract public that allows companies to extract public that allows companies to extract public web data at scale. web data at scale. web data at scale. And as we all know, public web data And as we all know, public web data And as we all know, public web data theoretically is available for everyone. theoretically is available for everyone. theoretically is available for everyone. But when you But in practice, if you But when you But in practice, if you But when you But in practice, if you want to connect your AI models, agents, want to connect your AI models, agents, want to connect your AI models, agents, databases, you need infrastructure databases, you need infrastructure databases, you need infrastructure layer. layer. layer. Uh so, this is what we do and this is Uh so, this is what we do and this is Uh so, this is what we do and this is where what matters now more than ever. where what matters now more than ever. where what matters now more than ever. Uh because the industry is shifting away Uh because the industry is shifting away Uh because the industry is shifting away from static knowledge and training from static knowledge and training from static knowledge and training itself still matters, of course. itself still matters, of course. itself still matters, of course. But training alone is no longer enough But training alone is no longer enough But training alone is no longer enough and to stay useful models needs to get and to stay useful models needs to get and to stay useful models needs to get access to fresh information, live access to fresh information, live access to fresh information, live search, real external data.

search, real external data. search, real external data. And without that, even the smartest And without that, even the smartest And without that, even the smartest model is limited by what it knows. model is limited by what it knows. model is limited by what it knows. And this is where my story begins. And this is where my story begins. And this is where my story begins. So, my name is Patricia and as I So, my name is Patricia and as I So, my name is Patricia and as I mentioned, I worked as uh in Oxylabs as mentioned, I worked as uh in Oxylabs as mentioned, I worked as uh in Oxylabs as a product manager now. But I started a product manager now. But I started a product manager now. But I started actually closer to engineering. I was actually closer to engineering. I was actually closer to engineering. I was leading teams dealing with service, core leading teams dealing with service, core leading teams dealing with service, core services. But the first squad that services. But the first squad that services. But the first squad that actually taught me one thing was what we actually taught me one thing was what we actually taught me one thing was what we called UX. called UX. called UX. Uh and what is UX? UX is usually means Uh and what is UX? UX is usually means Uh and what is UX? UX is usually means user experience. That is completely user experience. That is completely user experience. That is completely correct. But for us, that often meant correct. But for us, that often meant correct. But for us, that often meant closer to this, that client needs closer to this, that client needs closer to this, that client needs something really unusual. There is no something really unusual. There is no something really unusual. There is no ready-to-made product. The timeline is ready-to-made product. The timeline is ready-to-made product. The timeline is extremely painful, and somehow we need extremely painful, and somehow we need extremely painful, and somehow we need to build everything fast and make it to build everything fast and make it to build everything fast and make it work beautiful. work beautiful. work beautiful. So, the lesson that I learned with that So, the lesson that I learned with that So, the lesson that I learned with that team, that innovation never comes as a team, that innovation never comes as a team, that innovation never comes as a neat road map. It comes as a pressure, neat road map. It comes as a pressure, neat road map. It comes as a pressure, as a deadline, and sometimes, and quite as a deadline, and sometimes, and quite as a deadline, and sometimes, and quite often, as a trip report from San often, as a trip report from San often, as a trip report from San Francisco.

Francisco. Francisco. And this is how the first story started. And this is how the first story started. And this is how the first story started. One day, our sales team came back from One day, our sales team came back from One day, our sales team came back from San Francisco and said, "There is a San Francisco and said, "There is a San Francisco and said, "There is a demand for video API for AI training." demand for video API for AI training." demand for video API for AI training." Um there is one question that you Um there is one question that you Um there is one question that you actually really scared to ask the sales actually really scared to ask the sales actually really scared to ask the sales team. What's the deadline? team. What's the deadline? team. What's the deadline? Two weeks. What's the What's the scale? Two weeks. What's the What's the scale? Two weeks. What's the What's the scale? At least 5 petabytes per month. At least 5 petabytes per month. At least 5 petabytes per month. At that point, we have never built At that point, we have never built At that point, we have never built nothing like that, and it seems a lot. nothing like that, and it seems a lot. nothing like that, and it seems a lot. And actually, this is also a moment when And actually, this is also a moment when And actually, this is also a moment when the feature stops sounding less as a the feature stops sounding less as a the feature stops sounding less as a product feature. It sounds like product feature. It sounds like product feature. It sounds like infrastructure, because what client infrastructure, because what client infrastructure, because what client actually is asking to build is not just actually is asking to build is not just actually is asking to build is not just to download some videos. They are asking to download some videos. They are asking to download some videos. They are asking for a pipeline, collection, transfer, for a pipeline, collection, transfer, for a pipeline, collection, transfer, storage, delivery. storage, delivery. storage, delivery. And do it with enough reliability that And do it with enough reliability that And do it with enough reliability that would be compatible with AI training would be compatible with AI training would be compatible with AI training workloads. workloads. workloads. So, that story actually aged So, that story actually aged So, that story actually aged surprisingly well, because the market surprisingly well, because the market surprisingly well, because the market has moved exactly into that direction.

has moved exactly into that direction. has moved exactly into that direction. And AI infrastructure is becoming And AI infrastructure is becoming And AI infrastructure is becoming increasingly more multimodal. It's no increasingly more multimodal. It's no increasingly more multimodal. It's no longer about the text, and companies now longer about the text, and companies now longer about the text, and companies now need pipelines for video, metadata, need pipelines for video, metadata, need pipelines for video, metadata, transcripts, subtitles, and another transcripts, subtitles, and another transcripts, subtitles, and another structural context around the content structural context around the content structural context around the content itself. itself. itself. So, what with that? So, in two weeks, we So, what with that? So, in two weeks, we So, what with that? So, in two weeks, we had to build a new dedicated scraper had to build a new dedicated scraper had to build a new dedicated scraper with a brand new logic, new storage with a brand new logic, new storage with a brand new logic, new storage integrations, and with delivery flow of integrations, and with delivery flow of integrations, and with delivery flow of something that we actually never built something that we actually never built something that we actually never built before. before. before. And we actually made it, and somehow we And we actually made it, and somehow we And we actually made it, and somehow we made it even on time. But, this is not made it even on time. But, this is not made it even on time. But, this is not where the actually story ended. That was where the actually story ended. That was where the actually story ended. That was only the version one. only the version one. only the version one. So, So, So, client asked, "Great that you have a client asked, "Great that you have a client asked, "Great that you have a downloader, but what about transcripts?" downloader, but what about transcripts?" downloader, but what about transcripts?" So, we built a transcript support. So, we built a transcript support. So, we built a transcript support. Uh client tested out, and we see that Uh client tested out, and we see that Uh client tested out, and we see that all of the requests are failing. all of the requests are failing. all of the requests are failing. Then we start talking with the client, Then we start talking with the client, Then we start talking with the client, and we see that there is nothing that we and we see that there is nothing that we and we see that there is nothing that we did something wrong. That client did something wrong. That client did something wrong. That client actually didn't need a transcript, they actually didn't need a transcript, they actually didn't need a transcript, they needed the subtitles.

needed the subtitles. needed the subtitles. So, we adapt again. We build a subtitle So, we adapt again. We build a subtitle So, we adapt again. We build a subtitle support. support. support. Um Um Um then another request comes. "We are then another request comes. "We are then another request comes. "We are struggling to find videos in languages struggling to find videos in languages struggling to find videos in languages that we actually need. Can you build a that we actually need. Can you build a that we actually need. Can you build a search that we could gather those search that we could gather those search that we could gather those ideas?" ideas?" ideas?" So, we do it again. "What about So, we do it again. "What about So, we do it again. "What about metadata?" Of course, we do it once metadata?" Of course, we do it once metadata?" Of course, we do it once again. again. again. And this is the part of the story that I And this is the part of the story that I And this is the part of the story that I really loved, because once it started as really loved, because once it started as really loved, because once it started as a one product feature request, it a one product feature request, it a one product feature request, it actually became um actually became um actually became um became the whole product suite, because became the whole product suite, because became the whole product suite, because we started thinking that we're building we started thinking that we're building we started thinking that we're building just a downloader. Then we realized that just a downloader. Then we realized that just a downloader. Then we realized that we're building a transcript support, we're building a transcript support, we're building a transcript support, subtitle support, uh adding metadata, subtitle support, uh adding metadata, subtitle support, uh adding metadata, channel information, and ended up channel information, and ended up channel information, and ended up building our own internal library that building our own internal library that building our own internal library that glues everything together. glues everything together. glues everything together. And after enough iterations, And after enough iterations, And after enough iterations, uh as a one as I mentioned that started uh as a one as I mentioned that started uh as a one as I mentioned that started as a one-off time request, as a one-off time request, as a one-off time request, uh uh uh it became the product family. it became the product family. it became the product family. And in roughly 3 months, we actually And in roughly 3 months, we actually And in roughly 3 months, we actually ended up having the whole video API ended up having the whole video API ended up having the whole video API suite that supported downloaders, suite that supported downloaders, suite that supported downloaders, transcripts, subtitles, channel transcripts, subtitles, channel transcripts, subtitles, channel information. And yeah, after all of information. And yeah, after all of information. And yeah, after all of this, the final twist came.

this, the final twist came. this, the final twist came. So, it's 2026. So, it's 2026. So, it's 2026. Client already gathered 30 petabytes of Client already gathered 30 petabytes of Client already gathered 30 petabytes of data, and we're still waiting for a data, and we're still waiting for a data, and we're still waiting for a payment. payment. payment. So, yes, the first lesson is really So, yes, the first lesson is really So, yes, the first lesson is really technical, but also very human. technical, but also very human. technical, but also very human. Uh that innovation is actually a Uh that innovation is actually a Uh that innovation is actually a repeated adaptation under high pressure. repeated adaptation under high pressure. repeated adaptation under high pressure. Because once you learn that the client Because once you learn that the client Because once you learn that the client actually doesn't buy the first product actually doesn't buy the first product actually doesn't buy the first product iteration, they buy your ability to iteration, they buy your ability to iteration, they buy your ability to adapt. adapt. adapt. The next question becomes, can you The next question becomes, can you The next question becomes, can you actually make it under extreme latency actually make it under extreme latency actually make it under extreme latency constraints, too? constraints, too? constraints, too? And this is a part where I tell you a And this is a part where I tell you a And this is a part where I tell you a little about little about little about SERP data. SERP data. SERP data. And search data has always mattered. And search data has always mattered. And search data has always mattered. But the AI changed the role it plays. But the AI changed the role it plays. But the AI changed the role it plays. Um before, SERP was often used for Um before, SERP was often used for Um before, SERP was often used for analytics, SEO, monitoring, market analytics, SEO, monitoring, market analytics, SEO, monitoring, market intelligence. But now, it's a huge part intelligence. But now, it's a huge part intelligence. But now, it's a huge part of AI systems. It feeds retrieval of AI systems. It feeds retrieval of AI systems. It feeds retrieval pipelines. It grounds uh it powers pipelines. It grounds uh it powers pipelines. It grounds uh it powers assistance. It grounds answers. It helps assistance. It grounds answers. It helps assistance. It grounds answers. It helps agents interact with agents interact with agents interact with live information instead of stale live information instead of stale live information instead of stale training memory.

training memory. training memory. And that shift is not hypothetical. And that shift is not hypothetical. And that shift is not hypothetical. Google's grounding documentation Google's grounding documentation Google's grounding documentation explicitly positions Google Search as a explicitly positions Google Search as a explicitly positions Google Search as a way to connect models to current public way to connect models to current public way to connect models to current public knowledge. In simple terms, the model knowledge. In simple terms, the model knowledge. In simple terms, the model layer is increasingly expected to work layer is increasingly expected to work layer is increasingly expected to work with live retrieval layer around it. with live retrieval layer around it. with live retrieval layer around it. And that's why the next request mattered And that's why the next request mattered And that's why the next request mattered so much. so much. so much. So, back in 2024, client came and asked So, back in 2024, client came and asked So, back in 2024, client came and asked for SERP delivery with for SERP delivery with for SERP delivery with sub- sub-second SERP delivery. sub- sub-second SERP delivery. sub- sub-second SERP delivery. At that time, our traditional regular At that time, our traditional regular At that time, our traditional regular search scraper was around 4 seconds search scraper was around 4 seconds search scraper was around 4 seconds average latency. So, the gap was huge, average latency. So, the gap was huge, average latency. So, the gap was huge, but we still decided to go for it just but we still decided to go for it just but we still decided to go for it just to see if it's possible and we actually to see if it's possible and we actually to see if it's possible and we actually did it. did it. did it. But, the story doesn't have happy ending But, the story doesn't have happy ending But, the story doesn't have happy ending here because client did it not did not here because client did it not did not here because client did it not did not test it out. And to be honest, the test it out. And to be honest, the test it out. And to be honest, the market wasn't ready for that. So, we market wasn't ready for that. So, we market wasn't ready for that. So, we just put it on a shelf. just put it on a shelf. just put it on a shelf. But, what became clay clear later on But, what became clay clear later on But, what became clay clear later on that was never about making that was never about making that was never about making uh the old scraper faster because the uh the old scraper faster because the uh the old scraper faster because the regular scraper, what he does here is regular scraper, what he does here is regular scraper, what he does here is built to retrieve as much information as built to retrieve as much information as built to retrieve as much information as possible. So, we're talking ads, possible. So, we're talking ads, possible. So, we're talking ads, widgets, rich results, AI-generated widgets, rich results, AI-generated widgets, rich results, AI-generated results, different layouts.

results, different layouts. results, different layouts. And when we think about fast search API, And when we think about fast search API, And when we think about fast search API, it takes a different approach. It it takes a different approach. It it takes a different approach. It focuses on the things that actually focuses on the things that actually focuses on the things that actually matters only for AI systems. So, it's matters only for AI systems. So, it's matters only for AI systems. So, it's mostly organic results, top stories, mostly organic results, top stories, mostly organic results, top stories, news, and it cuts away all the heavy news, and it cuts away all the heavy news, and it cuts away all the heavy layout. layout. layout. So, even this small scope, it's already So, even this small scope, it's already So, even this small scope, it's already something to start something to start something to start thinking about lower latency. thinking about lower latency. thinking about lower latency. So, fast forward, it's 2025. So, fast forward, it's 2025. So, fast forward, it's 2025. Another client comes in and their Another client comes in and their Another client comes in and their request was simple: zero data retention, request was simple: zero data retention, request was simple: zero data retention, sub-second latency, sub-second latency, sub-second latency, and and and 2 weeks. 2 weeks. 2 weeks. For us, that meant For us, that meant For us, that meant to support different geolocation and to support different geolocation and to support different geolocation and query parameters, to have a system that query parameters, to have a system that query parameters, to have a system that is capable to deliver results under 800 is capable to deliver results under 800 is capable to deliver results under 800 milliseconds, and to have a solution milliseconds, and to have a solution milliseconds, and to have a solution that is ready uh to be tested out in that is ready uh to be tested out in that is ready uh to be tested out in less than 2 weeks. less than 2 weeks. less than 2 weeks. So, when your baseline is at 4 seconds, So, when your baseline is at 4 seconds, So, when your baseline is at 4 seconds, we are not talking about optimization. we are not talking about optimization. we are not talking about optimization. We are talking about redesign.

We are talking about redesign. We are talking about redesign. Uh so, we started from the scratch. Uh so, we started from the scratch. Uh so, we started from the scratch. And actually, the first version worked. And actually, the first version worked. And actually, the first version worked. In less than 2 In less than 2 In less than 2 weeks, we got around 650 milliseconds weeks, we got around 650 milliseconds weeks, we got around 650 milliseconds P90. So, that alone would be a great P90. So, that alone would be a great P90. So, that alone would be a great story, but the real story happened on story, but the real story happened on story, but the real story happened on the next call. the next call. the next call. So, we're sitting on a call with the So, we're sitting on a call with the So, we're sitting on a call with the client getting ready to test it out our client getting ready to test it out our client getting ready to test it out our new product. new product. new product. And while we were on the call, And while we were on the call, And while we were on the call, we got blocked. And we got blocked we got blocked. And we got blocked we got blocked. And we got blocked really bad. really bad. really bad. And to to be honest, this is really And to to be honest, this is really And to to be honest, this is really honest moment about when you think about honest moment about when you think about honest moment about when you think about infrastructure and systems, because this infrastructure and systems, because this infrastructure and systems, because this is a kind reminder that there is a is a kind reminder that there is a is a kind reminder that there is a difference between system that difference between system that difference between system that works in development, system that works works in development, system that works works in development, system that works in a test, and system that actually in a test, and system that actually in a test, and system that actually survives reality. survives reality. survives reality. So, we had to start over, because So, we had to start over, because So, we had to start over, because nothing worked. nothing worked. nothing worked. And at this second iteration was the And at this second iteration was the And at this second iteration was the hardest one, because we actually had to hardest one, because we actually had to hardest one, because we actually had to rely a lot on browsers.

rely a lot on browsers. rely a lot on browsers. And don't get me wrong, browsers are And don't get me wrong, browsers are And don't get me wrong, browsers are amazing. They are amazing. They are amazing. They are extremely useful, extremely useful, extremely useful, but browsers also are slow, expensive, but browsers also are slow, expensive, but browsers also are slow, expensive, complex, and deeply incompatible with complex, and deeply incompatible with complex, and deeply incompatible with dreams about low latency. dreams about low latency. dreams about low latency. So, So, So, there is So, we had a contradiction. there is So, we had a contradiction. there is So, we had a contradiction. The reality and the client wanted The reality and the client wanted The reality and the client wanted sub-second, the reality needed browsers, sub-second, the reality needed browsers, sub-second, the reality needed browsers, and browsers really wanted us to give us and browsers really wanted us to give us and browsers really wanted us to give us 4 seconds. 4 seconds. 4 seconds. So, at this point, So, at this point, So, at this point, there is no magic trick. You just go there is no magic trick. You just go there is no magic trick. You just go hunting for a time. So, you you review hunting for a time. So, you you review hunting for a time. So, you you review everything, layouts, parsers, sessions, everything, layouts, parsers, sessions, everything, layouts, parsers, sessions, proxies, every place when you can cut proxies, every place when you can cut proxies, every place when you can cut off a second, a two, a three, or four. off a second, a two, a three, or four. off a second, a two, a three, or four. And this is how systems becomes fast, And this is how systems becomes fast, And this is how systems becomes fast, not by giant breakthroughs as we thought not by giant breakthroughs as we thought not by giant breakthroughs as we thought at first, but by small decision that at first, but by small decision that at first, but by small decision that adds up. adds up. adds up. And that work paid off and actually And that work paid off and actually And that work paid off and actually evolved into something new. evolved into something new. evolved into something new. So, So, So, today we have fast search API that today we have fast search API that today we have fast search API that delivers results and fresh data directly delivers results and fresh data directly delivers results and fresh data directly into AI workflows with 550 milliseconds into AI workflows with 550 milliseconds into AI workflows with 550 milliseconds average latency.

average latency. average latency. And our scale move from 400 million And our scale move from 400 million And our scale move from 400 million daily requests to almost 6 billion daily daily requests to almost 6 billion daily daily requests to almost 6 billion daily requests. requests. requests. Uh so, that number matters. Uh so, that number matters. Uh so, that number matters. Because going from 400 million daily Because going from 400 million daily Because going from 400 million daily requests to 6 billion daily requests is requests to 6 billion daily requests is requests to 6 billion daily requests is not just a change, not just a growth. not just a change, not just a growth. not just a change, not just a growth. It's a change in operating model. It It's a change in operating model. It It's a change in operating model. It changes how you think about costs, changes how you think about costs, changes how you think about costs, observability, and failure of domains. observability, and failure of domains. observability, and failure of domains. So, the lesson of this part So, the lesson of this part So, the lesson of this part uh that in AI era, speed is not just uh that in AI era, speed is not just uh that in AI era, speed is not just performance. Speed actually defines what performance. Speed actually defines what performance. Speed actually defines what product can exist. Because in 4 seconds, product can exist. Because in 4 seconds, product can exist. Because in 4 seconds, you have a slow pipeline. In sub-second you have a slow pipeline. In sub-second you have a slow pipeline. In sub-second delivery, you have something that can delivery, you have something that can delivery, you have something that can sit and interact in your AI workflows. sit and interact in your AI workflows. sit and interact in your AI workflows. So, So, So, when a speed becomes product, when a speed becomes product, when a speed becomes product, what's next? what's next? what's next? Next is then scale actually becomes the Next is then scale actually becomes the Next is then scale actually becomes the real test. real test. real test. So, the first story was about So, the first story was about So, the first story was about mm adapting product scope. The second mm adapting product scope. The second mm adapting product scope. The second was adapting architecture for latency.

was adapting architecture for latency. was adapting architecture for latency. The third one is going to be adapting The third one is going to be adapting The third one is going to be adapting systems for scale. systems for scale. systems for scale. And the scale is where infrastructure And the scale is where infrastructure And the scale is where infrastructure becomes really humbling. becomes really humbling. becomes really humbling. At one point, another demand has forced At one point, another demand has forced At one point, another demand has forced us to scale our web and blocker quite us to scale our web and blocker quite us to scale our web and blocker quite aggressively. aggressively. aggressively. Uh I added just slides just to see how Uh I added just slides just to see how Uh I added just slides just to see how it works. it works. it works. Uh it's simple terms, it's similar to Uh it's simple terms, it's similar to Uh it's simple terms, it's similar to scraper, but it has proxy integration. scraper, but it has proxy integration. scraper, but it has proxy integration. So, So, So, we were working our way around 10,000 we were working our way around 10,000 we were working our way around 10,000 requests per second. Demand has forced requests per second. Demand has forced requests per second. Demand has forced us to scale us to scale us to scale to 60,000 requests per second to 60,000 requests per second to 60,000 requests per second and in less than 2 months. and in less than 2 months. and in less than 2 months. So, now that number alone sounds So, now that number alone sounds So, now that number alone sounds impressive, but it might be also impressive, but it might be also impressive, but it might be also misleading misleading if if you are misleading misleading if if you are misleading misleading if if you are thinking about as as a simple HTTP thinking about as as a simple HTTP thinking about as as a simple HTTP request. In our world, that means the request. In our world, that means the request. In our world, that means the end-to-end scraping job. Part uh it it end-to-end scraping job. Part uh it it end-to-end scraping job. Part uh it it will be routing, rendering, proxy will be routing, rendering, proxy will be routing, rendering, proxy handling, browsers execution, parsing, handling, browsers execution, parsing, handling, browsers execution, parsing, retries, normalization, and delivery retries, normalization, and delivery retries, normalization, and delivery itself. itself. itself. So, when you kind of scale to that So, when you kind of scale to that So, when you kind of scale to that workload, even adding up additional workload, even adding up additional workload, even adding up additional 2,000 servers doesn't solve the problem.

2,000 servers doesn't solve the problem. 2,000 servers doesn't solve the problem. You need an architecture. You need a You need an architecture. You need a You need an architecture. You need a central components that actually are central components that actually are central components that actually are reliable. You need observability that reliable. You need observability that reliable. You need observability that still tells you the truth. And you need still tells you the truth. And you need still tells you the truth. And you need testing that resembles testing that resembles testing that resembles ev- uh reality enough to matter. ev- uh reality enough to matter. ev- uh reality enough to matter. And this is where our main bottleneck And this is where our main bottleneck And this is where our main bottleneck showed up, not in dramatic outage, in showed up, not in dramatic outage, in showed up, not in dramatic outage, in load testing. load testing. load testing. Uh the hardest part was not generating Uh the hardest part was not generating Uh the hardest part was not generating synthetic traffic. Synthetic traffic is synthetic traffic. Synthetic traffic is synthetic traffic. Synthetic traffic is relatively easy comparing to reality. relatively easy comparing to reality. relatively easy comparing to reality. Uh but the hardest part, organic data Uh but the hardest part, organic data Uh but the hardest part, organic data testing. That means processing traffic testing. That means processing traffic testing. That means processing traffic that behave enough like real client that behave enough like real client that behave enough like real client usage to tell us something useful. usage to tell us something useful. usage to tell us something useful. And during one of those load tests, we And during one of those load tests, we And during one of those load tests, we hit the wall around 20,000 requests per hit the wall around 20,000 requests per hit the wall around 20,000 requests per second. second. second. At that point, there is no question if At that point, there is no question if At that point, there is no question if the system is actually working. It is the system is actually working. It is the system is actually working. It is working. The question becomes, do we working. The question becomes, do we working. The question becomes, do we actually know that it can go further? actually know that it can go further? actually know that it can go further? And that uncertainty was the real And that uncertainty was the real And that uncertainty was the real bottleneck. So, uh so are all the pain bottleneck. So, uh so are all the pain bottleneck. So, uh so are all the pain points, metrics, logs, and generating points, metrics, logs, and generating points, metrics, logs, and generating and processing everything at scale.

and processing everything at scale. and processing everything at scale. So, everybody loves observability in So, everybody loves observability in So, everybody loves observability in theory, but observability at scale theory, but observability at scale theory, but observability at scale becomes a true work because collecting becomes a true work because collecting becomes a true work because collecting logs is hard, processing logs is harder. logs is hard, processing logs is harder. logs is hard, processing logs is harder. Um Um Um and same applies to metrics. and same applies to metrics. and same applies to metrics. Uh Uh Uh they're essential, but when you scale up they're essential, but when you scale up they're essential, but when you scale up to that kind of a load, the telemetry to that kind of a load, the telemetry to that kind of a load, the telemetry itself becomes a part of the load and a itself becomes a part of the load and a itself becomes a part of the load and a part of the complexity. So, what we did? part of the complexity. So, what we did? part of the complexity. So, what we did? We scaled gradually. And eventually, we We scaled gradually. And eventually, we We scaled gradually. And eventually, we had to accept one unavoidable truth that had to accept one unavoidable truth that had to accept one unavoidable truth that the real testing is going to be with the real testing is going to be with the real testing is going to be with production traffic. And thankfully, that production traffic. And thankfully, that production traffic. And thankfully, that part actually went completely fine. Uh part actually went completely fine. Uh part actually went completely fine. Uh but the story doesn't end up here uh but the story doesn't end up here uh but the story doesn't end up here uh because because because the drama is still happening right now. the drama is still happening right now. the drama is still happening right now. Uh internally, we call this project 60 Uh internally, we call this project 60 Uh internally, we call this project 60 because we had to scale up to 60,000 because we had to scale up to 60,000 because we had to scale up to 60,000 requests per second. Now, requests per second. Now, requests per second. Now, it's already becoming project 150. So, it's already becoming project 150. So, it's already becoming project 150. So, while we were scaling our infrastructure while we were scaling our infrastructure while we were scaling our infrastructure to 60,000 requests per second, now we to 60,000 requests per second, now we to 60,000 requests per second, now we are talking and seeing results and scale are talking and seeing results and scale are talking and seeing results and scale up to about to about 1 100,000 requests up to about to about 1 100,000 requests up to about to about 1 100,000 requests per second.

per second. per second. So, the lesson from this part is also So, the lesson from this part is also So, the lesson from this part is also simple that the scale is never a finish simple that the scale is never a finish simple that the scale is never a finish line. Well, at least not for us. line. Well, at least not for us. line. Well, at least not for us. And probably when you reach one And probably when you reach one And probably when you reach one target number, the next one will appear. target number, the next one will appear. target number, the next one will appear. So, anyways, what does Oxylabs do in So, anyways, what does Oxylabs do in So, anyways, what does Oxylabs do in this whole this whole this whole thing? The I guess the stories make one thing? The I guess the stories make one thing? The I guess the stories make one thing quite clear that we are not just a thing quite clear that we are not just a thing quite clear that we are not just a proxy provider. Proxies are essential, proxy provider. Proxies are essential, proxy provider. Proxies are essential, they are important, but the hardest part they are important, but the hardest part they are important, but the hardest part and the larger job is building the and the larger job is building the and the larger job is building the infrastructure layer that allows infrastructure layer that allows infrastructure layer that allows companies to extract public web data up companies to extract public web data up companies to extract public web data up and and operate it at scale. That means and and operate it at scale. That means and and operate it at scale. That means reaching the open web, reaching the open web, reaching the open web, collecting data reliably, dealing with collecting data reliably, dealing with collecting data reliably, dealing with antibot systems, handling browsers when antibot systems, handling browsers when antibot systems, handling browsers when they are needed, instruction and they are needed, instruction and they are needed, instruction and delivering data, and doing in that delivering data, and doing in that delivering data, and doing in that manner that AI companies can actually manner that AI companies can actually manner that AI companies can actually plug into their systems. plug into their systems. plug into their systems. And this is exactly why it matters And this is exactly why it matters And this is exactly why it matters because the best thing we can offer is because the best thing we can offer is because the best thing we can offer is not just data access, not just data access, not just data access, it is this.

it is this. it is this. That you build the intelligence and we That you build the intelligence and we That you build the intelligence and we take the messy maintenance underneath take the messy maintenance underneath take the messy maintenance underneath because the messy part is is is real. because the messy part is is is real. because the messy part is is is real. The targets change, layouts change, The targets change, layouts change, The targets change, layouts change, detection changes, market itself detection changes, market itself detection changes, market itself changes, client needs changes. So, this changes, client needs changes. So, this changes, client needs changes. So, this is not a build-once business. This is an is not a build-once business. This is an is not a build-once business. This is an adapt-forever business. adapt-forever business. adapt-forever business. And honestly, that may be the most And honestly, that may be the most And honestly, that may be the most useful definition of innovation that I useful definition of innovation that I useful definition of innovation that I know. know. know. That innovation is the ability to keep That innovation is the ability to keep That innovation is the ability to keep adapting fast enough that a changing adapting fast enough that a changing adapting fast enough that a changing requirements becomes a new requirements becomes a new requirements becomes a new infrastructure. infrastructure. infrastructure. So, So, So, if I need you to leave with one thought if I need you to leave with one thought if I need you to leave with one thought today, I will probably get back where it today, I will probably get back where it today, I will probably get back where it started. started. started. That the next generation of AI will not That the next generation of AI will not That the next generation of AI will not be powered by better models. It will be be powered by better models. It will be be powered by better models. It will be powered by better infrastructure around powered by better infrastructure around powered by better infrastructure around it. it. it. Infrastructure that can connect models Infrastructure that can connect models Infrastructure that can connect models to reality, infrastructure that can to reality, infrastructure that can to reality, infrastructure that can push the web data directly into your push the web data directly into your push the web data directly into your pipelines, databases, agents, AI tools, pipelines, databases, agents, AI tools, pipelines, databases, agents, AI tools, infrastructure that can scale from 400 infrastructure that can scale from 400 infrastructure that can scale from 400 million daily requests to 6 billion million daily requests to 6 billion million daily requests to 6 billion daily requests.

daily requests. daily requests. Because this is really the story. Not Because this is really the story. Not Because this is really the story. Not just scale, not just scraping, not just just scale, not just scraping, not just just scale, not just scraping, not just speed, adaptation. Adapting products, speed, adaptation. Adapting products, speed, adaptation. Adapting products, adapting architecture, adapting systems. adapting architecture, adapting systems. adapting architecture, adapting systems. And doing it fast enough that the AI And doing it fast enough that the AI And doing it fast enough that the AI companies and you can keep on building companies and you can keep on building companies and you can keep on building while the maintenance burden stays with while the maintenance burden stays with while the maintenance burden stays with us. us. us. So, and this is what it actually means So, and this is what it actually means So, and this is what it actually means for me in AI world. It means that the for me in AI world. It means that the for me in AI world. It means that the model is not alone anymore. It already model is not alone anymore. It already model is not alone anymore. It already has bridge to it. Thank you.