Home / Use cases / Scrapy
Free Proxies for Scrapy
Scrapy has built-in proxy support through request.meta['proxy']. Combine it with a free, hourly-verified list and a small rotation middleware to spread requests across many IPs.
Rotating proxy middleware
Drop this middleware into your project and enable it in settings.py. It loads the verified list once, assigns a random proxy per request, and retries a different one when a proxy errors out.
# middlewares.py
import random, requests
class VerifiedProxyMiddleware:
LIST = "https://raw.githubusercontent.com/xyzs996/free-proxy-health-list/main/proxies/protocols/http/data.txt"
def __init__(self):
txt = requests.get(self.LIST, timeout=15).text
self.proxies = [f"http://{l.strip()}" for l in txt.splitlines() if l.strip()]
def process_request(self, request, spider):
request.meta["proxy"] = random.choice(self.proxies)
def process_exception(self, request, exception, spider):
request.meta["proxy"] = random.choice(self.proxies)
return request # retry with a different proxy
# settings.py
DOWNLOADER_MIDDLEWARES = {
"myproject.middlewares.VerifiedProxyMiddleware": 350,
"scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware": 400,
}
RETRY_ENABLED = True
RETRY_TIMES = 5
DOWNLOAD_TIMEOUT = 15
CONCURRENT_REQUESTS = 16
DOWNLOAD_DELAY = 1.0 # be polite
Tips
Free proxies are volatile, so keep RETRY_ENABLED on and give a generous RETRY_TIMES. Start from the fast subset for a better hit rate, and reload the list between long crawls since it refreshes hourly. Always respect the target site's robots.txt and terms.
Frequently asked questions
How do I set a single proxy in Scrapy?
Set request.meta['proxy'] = 'http://host:port', or export http_proxy. Scrapy's built-in HttpProxyMiddleware reads both.
Can I use SOCKS5 with Scrapy?
Scrapy's default middleware handles HTTP proxies; for SOCKS you need a third-party downloader handler. HTTP proxies from the HTTP list are the simplest path.
Where does the list come from?
Download the verified list as TXT or JSON from the CDN. It is re-checked and republished hourly, sorted fastest-first.