setting up rotation and sticky sessions correctly

a backconnect endpoint is one hostname and one port that resolves to a different exit every time you ask, or to the same exit for as long as you hold a token. both modes run on the same credentials, and you pick between them in the username rather than by connecting somewhere else.

the username is the control surface

http proxy auth carries a username and a password. providers overload the username with options, separated by hyphens, so a client that knows nothing except how to set proxy credentials can still steer country, city, network and session. the password never changes.

# rotating: a new exit is chosen per request
curl --proxy "http://$USER:$PASS@$HOST:$PORT" https://api.ipify.org

# pinned: the same exit answers while the token lives
curl --proxy "http://$USER-session-ab12:$PASS@$HOST:$PORT" https://api.ipify.org
curl --proxy "http://$USER-session-ab12:$PASS@$HOST:$PORT" https://api.ipify.org

# pinned and placed: a german exit, held under one token
curl --proxy "http://$USER-country-de-session-ab12:$PASS@$HOST:$PORT" https://api.ipify.org

rotate what is stateless, pin what is not

rotation spreads volume so that no single address collects enough requests to cross a rate limit or attract a reputation hit. it breaks anything the target tracks per address, and it breaks it silently.

• signing in, and every request after the sign in.
• a cart, a checkout, or any multi step form carrying a csrf token.
• pagination that uses a server side cursor rather than a page number.
• anything that set a cookie you are expected to send back.
• a captcha you already passed, since the pass is scored against the address that earned it.

the rest of an ordinary scrape is stateless: independent urls, no sign in, no cursor. rotate those freely and pace them per exit.

a session token is an identity, so spend it deliberately

the token maps to an exit for as long as the provider holds the mapping, usually a few minutes to an hour. changing the token moves you to a fresh exit, which is how you keep one identity separate from the next. reusing one token across two accounts links those accounts through the address, which is the mistake people make while trying to be careful.

import requests

USER = "your-account"
PASS = "your-proxy-password"
HOST, PORT = "gateway.example.net", 8080


def proxies_for(session=None, country=None):
    """options ride in the username, so one endpoint serves every mode."""
    parts = [USER]
    if country:
        parts += ["country", country]
    if session:
        parts += ["session", session]
    url = f"http://{'-'.join(parts)}:{PASS}@{HOST}:{PORT}"
    return {"http": url, "https": url}


# one identity: sign in and hold the exit for the whole flow
s = requests.Session()
s.proxies = proxies_for(session="cart-7f3a", country="us")
s.post("https://target.example/login", data={"u": "...", "p": "..."}, timeout=30)
cart = s.get("https://target.example/cart", timeout=30)
print(cart.status_code)

plan the failure, because the exit will drop

a residential or mobile exit is a real device on a real line. it can go offline in the middle of your request, and a pinned session can come back on a different address without announcing it. treat stickiness as a strong hint rather than a guarantee, and write code that notices when the address moved.

1. retry on a fresh token rather than the same one, since the address you just used is the thing that failed.
2. back off between attempts, because a burst of retries damages the reputation of every exit it touches.
3. read 429 as pacing information rather than an error, and honour retry-after when the response carries it.
4. check that the body is the page you asked for. a 200 carrying a consent wall is a failure with a success code.
5. cap the attempts and record the url, so a permanently blocked target shows up as a number instead of a loop.

import itertools
import random
import time

import requests

_tokens = itertools.count()


def fetch(url, attempts=4):
    """each attempt takes a new exit, with a widening pause between tries."""
    last = None
    for n in range(attempts):
        token = f"job-{next(_tokens)}"
        try:
            r = requests.get(url, proxies=proxies_for(session=token), timeout=30)
            if r.status_code == 429:
                time.sleep(float(r.headers.get("retry-after", 2 ** n)))
                continue
            r.raise_for_status()
            return r.text
        except requests.RequestException as exc:
            last = exc
            time.sleep(2 ** n + random.random())
    raise RuntimeError(f"{url}: gave up after {attempts} attempts ({last})")

pace per exit, not globally

a rate limit is normally keyed on the source address, so the number to tune is requests per second per exit. fifty parallel requests through one address is a burst. the same fifty spread over fifty addresses is unremarkable traffic. raising global concurrency without adding exits moves you towards the first shape and buys timeouts rather than throughput.

if the limit is keyed on your signed in account instead, more addresses change nothing and make the account look stranger than it did. slow down there, and accept that a pool is not the tool for that particular wall.