Home / Use cases / Web scraping

Free Proxies for Web Scraping

Scraping a site from a single IP gets you rate-limited or blocked fast. Rotating through a fresh, verified proxy list spreads your requests across many IPs — here is how to do it with a free, hourly-checked list and no signup.

Why rotate proxies

Most sites throttle or ban an IP that sends too many requests. A proxy pool lets each request (or each batch) go out from a different address, so no single IP trips the rate limit. The catch with free proxies is that any individual one can die within minutes — so the reliable pattern is rotate + retry over many verified proxies, not depend on one.

Rotate a verified list in Python

This example loads the JSON list (already sorted fastest-first), then rotates through it, retrying the next proxy whenever one fails.

import requests

LIST = "https://raw.githubusercontent.com/xyzs996/free-proxy-health-list/main/proxies/all/data.json"

# 1. Load the verified list (fastest, highest quality first)
pool = requests.get(LIST, timeout=15).json()

def scheme(p):
    proto = p["protocol"]
    host = p["proxy"]
    return f"socks5h://{host}" if proto.startswith("socks5") else \
           f"socks4://{host}" if proto.startswith("socks4") else f"http://{host}"

def fetch(url):
    for p in pool:                       # 2. rotate through the pool
        px = scheme(p)
        try:
            r = requests.get(url, proxies={"http": px, "https": px}, timeout=8)
            if r.ok:
                return r.text
        except requests.RequestException:
            continue                     # 3. dead proxy? try the next one
    raise RuntimeError("all proxies failed — refresh the list")

print(fetch("https://httpbin.org/ip")[:200])

Do it politely

1. Add a delay with jitter between requests (e.g. 1–5s) so you don't hammer the target.
2. Set a realistic User-Agent header.
3. Honor robots.txt and the site's terms of service.
4. Back off (sleep longer) when you see 429 Too Many Requests.
5. Refresh the proxy list periodically — it is re-verified hourly.

Only scrape data you are allowed to. Do not use proxies to bypass access controls, log in on someone's behalf, or violate a site's terms. See the responsible-use note.

Frequently asked questions