1. middlewares.py

import itertools, random

class RotatingProxyMiddleware:
    def __init__(self):
        with open("proxies.txt") as handle:
            self.proxies = [line.strip() for line in handle if line.strip()]
        random.shuffle(self.proxies)
        self.pool = itertools.cycle(self.proxies)

    def process_request(self, request, spider):
        request.meta["proxy"] = f"http://{next(self.pool)}"

2. settings.py

DOWNLOADER_MIDDLEWARES = {
    "myproject.middlewares.RotatingProxyMiddleware": 350,
    "scrapy.downloadermiddlewares.retry.RetryMiddleware": 550,
}
RETRY_TIMES = 5
DOWNLOAD_TIMEOUT = 10

Notes that save time

Order matters: the proxy middleware must run before the retry middleware so each retry picks a fresh proxy rather than repeating the failed one. Keep DOWNLOAD_TIMEOUT low — free proxies are the slowest part of the pipeline — and raise RETRY_TIMES to absorb the failure rate.

Public snapshot boundary
Free public proxies are shared and operated by unknown parties. Never send passwords, tokens or personal data through them. Follow the GitHub Acceptable Use Policies: no spam, no attacks, no bypassing access controls, no scraping against site policies.

Scrapy proxy middleware

1. middlewares.py

2. settings.py

Notes that save time

Related guides