metered proxies bill by the gigabyte, and it is not the gigabyte your parser sees. the gap between the two is where surprise invoices come from, and all of it is explainable.
the meter counts bytes on the wire, in both directions
a gateway relays a stream. as each chunk arrives the meter adds its length to a counter and the same bytes pass straight on, so what gets counted is what crossed the socket: request headers, tls records, the response body as it travelled, the redirects you followed, and everything you downloaded and threw away. upload counts too, which matters for posts and file uploads and for very little else.
the same figures are visible from the client side, which is the honest way to size a job before running it at scale.
# what one fetch actually costs on the wire
curl --proxy "$PROXY" -sS -o /dev/null \
--write-out 'up=%{size_upload} down=%{size_download} headers=%{size_header} secs=%{time_total}\n' \
"https://target.example/api/items?page=1"multiply by the number of pages, add a margin for retries, and you have a figure worth comparing against a plan before committing to it.
why your parser sees a different number
- compression. the meter counts the compressed bytes that crossed the socket, and your parser sees the text after expansion, so the page looks bigger than it billed.
- overhead. headers, tls records and connection setup are on the wire and never reach your parse step.
- failure. redirects, retries, timeouts and 403s all move bytes, and not one of them produces a row.
- assets. a browser pulls images, fonts, css, video and third party scripts your extractor never touches.
- waste. fetching a url you already have costs exactly what fetching a new one costs.
the levers that actually move the total
- block images, media and fonts when you drive a browser. this normally cuts more than every other change combined.
- call the json endpoint the page calls, rather than rendering the page to read the same data.
- keep compression on, and send an accept-encoding header that says so.
- cache by url, and never fetch the same one twice in a run.
- cap redirect following, so a chain into a heavy asset stops early.
- stop retrying urls that have failed the same way three times.
from playwright.sync_api import sync_playwright
# a listing page without these is a fraction of the bytes and the same html
DROP = {"image", "media", "font", "stylesheet"}
with sync_playwright() as p:
browser = p.chromium.launch(proxy={
"server": "http://gateway.example.net:8080",
"username": "your-account",
"password": "your-proxy-password",
})
page = browser.new_page()
page.route(
"**/*",
lambda route: route.abort()
if route.request.resource_type in DROP
else route.continue_(),
)
page.goto("https://target.example/listing", wait_until="domcontentloaded")
print(page.title())
browser.close()metered against flat
a flat product is a line rather than a meter: a fixed speed for a fixed period, with the bytes uncounted. the comparison is arithmetic. take your monthly gigabytes, price them at the tier your volume falls in, and set that against the flat line for the same period. below the crossover the meter wins and above it the line does.
two things distort that arithmetic in practice. a flat line is a small number of addresses, so it cannot spread a crawl the way a pool can. and a meter has no ceiling, so a bug that loops on a heavy page spends real money quietly. a hard cap on the account is cheap insurance against the second.
measure before you tune
record bytes per successful row, not bytes per day. the first is a number you can act on and compare between runs. the second moves with volume and hides every regression inside it. a change that halves bytes per row is worth more than any negotiation over a rate.
check the figure again after the target changes its pages. a redesign that adds a video header to a product page can double the cost of a scrape overnight without one line of your code changing.