Proxies for Web Scraping
Spread crawler traffic across a pool of static datacenter IPs so no single address carries the whole load.
What web scraping needs from a proxy
Check the site’s terms, robots.txt and any official API or data feed first: when one fits the job, it is usually more reliable than scraping. If your permitted crawler then reaches a site’s per-IP limits, a proxy pool can spread its requests across more addresses. Proxies do not change what a site allows, and many crawls never need them. The right choice depends on the target: shared pools suit tolerant public-data sources, while dedicated IPs are worth it when a site is strict about shared or recycled addresses.
- A pool that fits your volume
- More IPs means fewer requests per IP for the same total. Size the plan to requests per hour, not total records, and stay within the limits each site sets.
- IP reputation
- IPs with no recent abuse history get through more often. A dedicated allocation keeps other customers off your IPs, though it cannot erase an address’s past.
- The right country
- Geo-locked content needs an IP in that country. Spanish and Italian IPs reach EU-only content; US IPs reach US-only data.
- Flexible authentication
- IP whitelist suits fixed crawler machines; username and password suits cloud runners whose source IP changes.
- Concurrency
- Scrapers often run hundreds of requests in parallel. Both the network and the pool size have to keep up.
- HTTPS support
- Almost every site is HTTPS. The proxy tunnels TLS end to end with CONNECT, so certificates stay intact.
A working starting point
Python, rotating through your proxy list. Replace the placeholder login and IP with your own from the control panel, or whitelist your machine's IP and drop the login.
New to proxy setup? Start with the HTTP proxy setup guide.
import itertools, requests
# One line per proxy, exported from the control panel as ip:port:user:pass
lines = open("proxies.txt").read().split()
pool = itertools.cycle(
f"http://{u}:{p}@{ip}:{port}" for ip, port, u, p in (l.split(":") for l in lines)
)
for url in ["https://example.com/page/1", "https://example.com/page/2"]:
proxy = next(pool)
r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=20)
print(r.status_code, url)Which line to buy
In order of fit for web scraping.
- USUSA Shared ProxiesThe lowest price per IP for high-volume scraping of US public data, link checks and large crawls where IP history is not the limiting factor.
- USUSA Private ProxiesDedicated IPs for targets that are strict about shared pools, such as many e-commerce, ticketing and social platforms.
- ESSpanish Datacenter ProxiesSpanish IPs for Spain-only content, Spanish marketplaces and anything that changes by Spanish location.
Web Scraping questions
How many proxies do I need for my scraper?
There is no universal number. As an illustrative starting point only: dividing your peak requests per minute by 6 gives the IP count at which each IP averages one request every 10 seconds. That is not a safe rate for any particular site. Run a small test within the site’s rules, measure how many requests succeed and what limits it applies, and adjust from there. Send sales your numbers if you want help sizing a plan.
Do these work with Scrapy, Playwright or Puppeteer?
Yes. They are standard HTTP(S) proxies and work with any client that accepts a proxy URL. Use IP whitelist or username and password, whichever suits your runtime.
Will these work on Cloudflare or DataDome protected sites?
It depends on the site and how it is configured. Some protected sites accept datacenter IPs and others reject them, whichever provider you use; residential IPs are sometimes accepted where datacenter ranges are not. Ask sales about your target before ordering.
Datacenter or residential: which do I need?
Datacenter for most scraping: faster, cheaper and effective on most targets. Residential if your target blocks datacenter networks outright. We sell both; residential is quoted.
Can I rotate IPs automatically?
Datacenter IPs are static and do not rotate on their own. Most scraping frameworks rotate client-side by cycling through your proxy list, as in the example above. Monthly refreshes are available on request.
Other use cases
- SEO MonitoringCheck rankings from the country you are tracking, with static IPs that keep comparisons repeatable.
- Price MonitoringCheck retailer prices from US, Spanish or Italian IPs, with the rest of each session set up for that market.
- Account ManagementOne consistent, dedicated IP per account, in the country that fits the account.
- Ad VerificationLoad your placements as a visitor in each market you bought, and record what actually served.
Not sure which plan fits your workload? Tell us about it and we will recommend one.