Why Headless Scraping Fails and Headful Playwright Doesn't
- Source
- x.com
- Date
✨ Okay with lots of help from @javilopen and his Spanish scraping friends I've managed to vibe code my own @Scrapingbee alternative and replace my $249/mo Scrapingbee plan with my own $1/mo scraper running on my VPS! First I tried OpenSERP, an open source project that seems freshly maintained but it didn't work on my VPS IP or laptop IP and got blocked fast Then I tried to use residential IP proxies from IPRoyal but they were already quite tainted IPs meaning Google already knew they were scrapers and just blocked about 60% of them already, not so great, also didn't work! Then I discovered OpenSERP uses a vanilla headless Chrome, Google instantly detects that and blocks that, so literally the entire repo is useless cause it won't even work? I don't get why they don't say that in the docs? OpenSERP is even worse than just basic CURL, as my AI said: "plain curl got 7% blocked but OpenSERP's headless browser got 100% blocked" So then I just built my own with Playwright and a real Google Chrome on my VPS, headful and persistent context, and slow pacing like every 30 seconds or so And I got some better residential IP recommendations (which I can't tell you because thenhey t stop…
Context
Indie developer levelsio says he replaced his $249/month Scrapingbee subscription with a self-built scraper costing about $1/month. He first tried OpenSERP, an open-source scraping tool, but reports it got blocked immediately because it runs a vanilla headless Chrome browser, a stripped-down browser instance without a visible window, which he says sites like Google can detect and block outright. He also tried routing requests through residential IP proxies from IPRoyal, but says roughly 60% of those IPs were already flagged as scrapers before he started.
What worked, according to his account, was building his own scraper with Playwright, a browser automation tool, running a real, visible (headful) Chrome instance with a persistent browsing session and deliberately slow pacing, about one request every 30 seconds. He reports around a 90% success rate, similar to or better than Scrapingbee's paid service. These are his own self-reported results from one setup, not an independently verified .
- benchmark — A standard public test set for comparing AI models — the shared scoreboards behind every "model X beats model Y" claim.
- context window — The maximum amount of text a model can consider at once — its working memory for the current conversation or task.
“plain curl got 7% blocked but OpenSERP's headless browser got 100% blocked”
“OpenSERP is even worse than just basic CURL, as my AI said”
“Now I'm at about 90% success rate, which is similar or better than Scrapingbee”
Checking sign-in…
Loading comments…