How Web Data Infrastructure Powers the Next Generation of AI — Patricija Žemaitytė, Oxylabs

Fast web-data retrieval and dependable web-data retrieval are separate engineering goals. Web-data provider Oxylabs reports reducing a search API's average response time from four seconds to 550 milliseconds, while also recounting a client demonstration that was blocked by the target service. Both appear in its infrastructure talk.

The speed work involved changes across layouts, parsing, sessions and proxies. The blocking incident illustrates why that work alone does not establish reliable delivery of information to an AI system. A retrieval service must remain usable under the conditions imposed by the websites it accesses, even after its successful requests become much faster.

“Patricija Žemaitytė treats that as the useful distinction: something that works in development, something that passes tests, and something that survives reality are three different systems.”

“It averages 550 milliseconds now, against a 4 second baseline.”

“The punchline she offers is that the client has since collected 30 petabytes and has not paid yet.”

“Scaling the unblocker from 10,000 to 60,000 requests per second hit a wall around 20,000 in load testing, where the real difficulty was not generating synthetic traffic but knowing whether the number meant anything, since telemetry at that volume becomes part of the load it measures.”

“Her argument throughout is that this is not a build once business, it is an adapt forever one.”