proxies for ai and llm training data collection
building a dataset is a crawl measured in millions of pages. two things decide whether it finishes: how fast your addresses get blocked, and what the bandwidth costs.
why this needs proxies
a large crawl from a narrow range of addresses is throttled or blocked long before it completes, and on rendering-heavy pages the bandwidth bill, not the request count, becomes the dominant cost. spreading the crawl across real residential addresses keeps it running, and choosing the right pool for each source is what keeps the bill in proportion to the data.
which proxies to use
what it costs
estimate: about 0.5 mb per page for html plus light assets. crawls that render each page in a full browser run several times higher, which is why the pool choice matters at this scale.
| pages | bandwidth | residential lite | datacenter | residential |
|---|---|---|---|---|
| 1,000,000 | 488 gb | $122.00 | $219.60 | $780.80 |
| 10,000,000 | 4,883 gb | $976.60 | $1709.05 | $6347.90 |
| 50,000,000 | 24,414 gb | $4882.80 | $8544.90 | $31738.20 |
volume rates are applied automatically, so the larger rows already reflect the cheaper tier.
how to set it up
faq
- residential or datacenter for large crawls?
- start on datacenter for sources that do not check ip reputation, since it is far cheaper and faster. move a source to residential the moment datacenter addresses start getting blocked on it.
- what drives the cost of a big crawl?
- bandwidth, once you are at scale. request count matters less than average page weight, so scraping raw html instead of rendering a full browser is the single biggest lever on the bill.
- how do i keep a crawl from getting blocked?
- spread it. rotate across enough addresses that no single one is requesting faster than a person plausibly would, and match the pool to how strictly each source checks reputation.