1. Rotate with failure tracking
import collections, itertools, requests
proxies = [l.strip() for l in open("proxies.txt") if l.strip()]
failures = collections.Counter()
pool = itertools.cycle(proxies)
def get(url, attempts=10):
for _ in range(attempts):
proxy = next(pool)
if failures[proxy] >= 3:
continue
try:
return requests.get(
url,
proxies={"http": f"http://{proxy}", "https": f"http://{proxy}"},
timeout=8,
)
except requests.RequestException:
failures[proxy] += 1
raise RuntimeError("pool exhausted")Notes that save time
Counting failures per proxy and skipping repeat offenders is what turns a raw list into a usable pool — without it you keep paying the timeout for the same dead entries. Refresh the file on the publish cadence and reset the counter when you do. For production workloads where you cannot absorb this failure rate, a maintained rotating endpoint is the better tool.
Related guides
Public snapshot boundary
Free public proxies are shared and operated by unknown parties. Never send passwords, tokens or personal data through them. Follow the GitHub Acceptable Use Policies: no spam, no attacks, no bypassing access controls, no scraping against site policies.
Free public proxies are shared and operated by unknown parties. Never send passwords, tokens or personal data through them. Follow the GitHub Acceptable Use Policies: no spam, no attacks, no bypassing access controls, no scraping against site policies.