blocked is not a diagnosis. a request can be refused by the network the address belongs to, by the rate you sent it at, by the tls handshake your client makes, or by html you never read. each has a different fix, and buying different proxies is the correct answer to exactly one of them.
read the refusal first
| what you get | usual cause | what to change |
|---|---|---|
| 403 on the very first request | the address was classified before you did anything | the exit type, not the headers |
| 429, sometimes with retry-after | a rate limit keyed on the exit address | pacing per exit, and more exits |
| a captcha or a challenge page | a low score assembled from the address and the client together | both, in that order |
| a reset during the tls handshake | the client hello was rejected before any http was sent | the client library |
| 200 with thin, stale or empty data | quiet degradation, which is a block that does not announce itself | nothing, until you compare against a direct fetch |
the last row is the one that costs money. a scraper storing nulls for a week looks healthy in every metric except the data.
isolate the layer
three requests, in this order, tell you which layer to work on. run them by hand before touching any code.
is the proxy itself healthy
# ipify answers with the exit address, and has no reason to refuse anyone,
# so a failure here belongs to the proxy rather than to your target
curl --proxy "$PROXY" --silent --show-error \
--write-out '\nstatus=%{http_code} time=%{time_total}s\n' \
https://api.ipify.orgif that fails, the cause is credentials, the endpoint, or the pool having nothing available for the filters you asked for. narrow filters are the common one, and dropping city or asn is the quickest way to prove it.
does the target refuse the exit, or the client
# the same request twice: through the proxy, then straight from this machine.
# headers only, so you see the status and the defences without pulling a body.
TARGET="https://target.example/product/1234"
curl --proxy "$PROXY" -sS -D - -o /dev/null "$TARGET"
curl -sS -D - -o /dev/null "$TARGET"if both are refused, your client is the problem and a different exit will not help. if only the proxied one is refused, the address is the problem. if both succeed, the difference lives in your code: the headers you send, the order you send them in, or the rate.
does a real browser through the same exit succeed
point a headless browser at the target using the same proxy credentials. a browser that gets through where your http client did not is telling you the refusal was about the handshake and the javascript, not about the address. that moves the work to your client and saves you from buying a pool that would not have helped.
the usual causes, in the order they occur
- the exit sits on a hosting asn and the target refuses hosting ranges outright.
- the exit is fine but its neighbours are not, because reputation is often aggregated across the surrounding block.
- your tls handshake names one client while your user agent header names another.
- your header set is short, oddly ordered, or missing things a browser always sends.
- you are pacing globally rather than per exit, so one address absorbs the whole burst.
- you rotated in the middle of something stateful and lost a session you had already paid for.
- the site changed, and you are parsing a consent page instead of a product page.
change one thing at a time
blocks feel unsolvable because people change the pool, the headers and the rate together and then cannot say which of the three mattered. hold a single target url and a fixed request, vary one input per run, and record the status and the response length every time. the length is what catches the quiet failures.
when you do change the pool, give the new one enough requests to show its behaviour. the first request from a fresh address is the one most likely to succeed, so a small sample flatters whatever you tried last.