Proxies for Web Scraping

Spread crawler traffic across a pool of static datacenter IPs so no single address carries the whole load.

What web scraping needs from a proxy

Check the site’s terms, robots.txt and any official API or data feed first: when one fits the job, it is usually more reliable than scraping. If your permitted crawler then reaches a site’s per-IP limits, a proxy pool can spread its requests across more addresses. Proxies do not change what a site allows, and many crawls never need them. The right choice depends on the target: shared pools suit tolerant public-data sources, while dedicated IPs are worth it when a site is strict about shared or recycled addresses.

A pool that fits your volume
More IPs means fewer requests per IP for the same total. Size the plan to requests per hour, not total records, and stay within the limits each site sets.
IP reputation
IPs with no recent abuse history get through more often. A dedicated allocation keeps other customers off your IPs, though it cannot erase an address’s past.
The right country
Geo-locked content needs an IP in that country. Spanish and Italian IPs reach EU-only content; US IPs reach US-only data.
Flexible authentication
IP whitelist suits fixed crawler machines; username and password suits cloud runners whose source IP changes.
Concurrency
Scrapers often run hundreds of requests in parallel. Both the network and the pool size have to keep up.
HTTPS support
Almost every site is HTTPS. The proxy tunnels TLS end to end with CONNECT, so certificates stay intact.

A working starting point

Python, rotating through your proxy list. Replace the placeholder login and IP with your own from the control panel, or whitelist your machine's IP and drop the login.

New to proxy setup? Start with the HTTP proxy setup guide.

Python
import itertools, requests

# One line per proxy, exported from the control panel as ip:port:user:pass
lines = open("proxies.txt").read().split()
pool = itertools.cycle(
    f"http://{u}:{p}@{ip}:{port}" for ip, port, u, p in (l.split(":") for l in lines)
)

for url in ["https://example.com/page/1", "https://example.com/page/2"]:
    proxy = next(pool)
    r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=20)
    print(r.status_code, url)
Placeholder login and a documentation IP. Use your own from the control panel.

Web Scraping questions

How many proxies do I need for my scraper?

There is no universal number. As an illustrative starting point only: dividing your peak requests per minute by 6 gives the IP count at which each IP averages one request every 10 seconds. That is not a safe rate for any particular site. Run a small test within the site’s rules, measure how many requests succeed and what limits it applies, and adjust from there. Send sales your numbers if you want help sizing a plan.

Do these work with Scrapy, Playwright or Puppeteer?

Yes. They are standard HTTP(S) proxies and work with any client that accepts a proxy URL. Use IP whitelist or username and password, whichever suits your runtime.

Will these work on Cloudflare or DataDome protected sites?

It depends on the site and how it is configured. Some protected sites accept datacenter IPs and others reject them, whichever provider you use; residential IPs are sometimes accepted where datacenter ranges are not. Ask sales about your target before ordering.

Datacenter or residential: which do I need?

Datacenter for most scraping: faster, cheaper and effective on most targets. Residential if your target blocks datacenter networks outright. We sell both; residential is quoted.

Can I rotate IPs automatically?

Datacenter IPs are static and do not rotate on their own. Most scraping frameworks rotate client-side by cycling through your proxy list, as in the example above. Monthly refreshes are available on request.

Not sure which plan fits your workload? Tell us about it and we will recommend one.