proxies for web scraping
scraping at volume fails on two things: your ip getting blocked, and the bill for the bandwidth it took. this is how to size both before you start.
why this needs proxies
a scraper making thousands of requests from one address is trivially detectable, and most targets rate-limit or block on ip long before they look at anything else. rotating residential addresses spread the requests across real consumer ips, so each one looks like an ordinary visitor.
which proxies to use
residential from $1.30
the default for targets that check ip reputation. rotating, 160+ countries, city and asn targeting when the job needs a specific market.
residential lite from $0.20
the same residential behaviour at a lower rate, for high-volume jobs where a slightly smaller pool is an acceptable trade.
datacenter from $0.35
only when the target does not check ip reputation. much cheaper per gb and much faster, but blocked quickly by anything that does.
what it costs
estimate: about 0.5 mb per page, counting html and the assets a scraper usually pulls. text-only scraping runs well under this; anything rendering a full browser runs over.
| pages | bandwidth | residential | residential lite | datacenter |
|---|---|---|---|---|
| 10,000 | 5 gb | $9.75 | $1.50 | $3.00 |
| 100,000 | 49 gb | $95.55 | $14.70 | $29.40 |
| 1,000,000 | 488 gb | $780.80 | $122.00 | $219.60 |
volume rates are applied automatically, so the larger rows already reflect the cheaper tier.
how to set it up
sessionrotating, so every request leaves from a different address
accountone random 16-digit number. no email, no name, no kyc.
paymentcrypto only: monero, bitcoin, litecoin and ethereum.
faq
- how much bandwidth does scraping actually use?
- it depends almost entirely on whether you render the page. raw html is often 50-200 kb; a full headless browser pulling images, css and scripts is frequently 2-5 mb. the estimate on this page assumes about 0.5 mb per page, which is typical for html plus light assets.
- residential or datacenter for scraping?
- datacenter is far cheaper per gb and faster, and works fine on targets that do not check ip reputation. the moment a target does, datacenter ips get blocked quickly and residential is the only thing that keeps working.
- do i need a new proxy for every request?
- with a rotating session you get a different exit ip per request automatically, which is usually what a scraper wants. use a sticky session when a target requires the same ip across a multi-step flow such as a login or a cart.