Configuring proxies in Python (Requests, Selenium, Scrapy)

This guide walks through exactly how to configure proxies with three of the most common Python tools: Requests, Selenium, and Scrapy.

1. Using Proxies with Python Requests

The requests library is one of the simplest ways to make HTTP requests in Python. Here's how to route those requests through a proxy.

Basic Setup

import requests

proxies = {
    "http": "http://proxy_ip:port",
    "https": "http://proxy_ip:port",
}

response = requests.get("https://api.ipify.org", proxies=proxies, timeout=10)
print("Your IP is:", response.text)

Rotating Proxies

If you have a list of proxy IPs, you can rotate them manually:

import random


proxy_list = [
    "http://proxy1:port",
    "http://proxy2:port",
    "http://proxy3:port",
]

proxy = random.choice(proxy_list)
proxies = {
    "http": proxy,
    "https": proxy,
}

response = requests.get("https://httpbin.org/ip", proxies=proxies)
print(response.json())

Handling Errors Gracefully

try:

   response = requests.get("https://example.com", proxies=proxies, timeout=10)
    response.raise_for_status()
    response.raise_for_status()
except requests.exceptions.RequestException as e:
    print("Request failed:", e) 

2. Using Proxies with Selenium (Headless Browsers)

Selenium is often used for JavaScript-heavy websites. Here’s how to configure it to run with proxies.

Chrome Example

from selenium import webdriver
from selenium.webdriver.chrome.options import Options
from selenium.webdriver.chrome.options import Options

proxy = "proxy_ip:port"

chrome_options = Options()
chrome_options.add_argument(f'--proxy-server=http://{proxy}')

driver = webdriver.Chrome(options=chrome_options)
driver.get("https://api.ipify.org")
print(driver.page_source)
driver.quit()

Firefox Example

from selenium import webdriver
from selenium.webdriver.firefox.options import Options

proxy = "proxy_ip:port"

firefox_profile = webdriver.FirefoxProfile()
firefox_profile.set_preference("network.proxy.type", 1)
firefox_profile.set_preference("network.proxy.http", "proxy_ip")
firefox_profile.set_preference("network.proxy.http_port", int("port"))
firefox_profile.set_preference("network.proxy.ssl", "proxy_ip")
firefox_profile.set_preference("network.proxy.ssl_port", int("port"))
firefox_profile.update_preferences()

options = Options()
driver = webdriver.Firefox(firefox_profile=firefox_profile, options=options)
driver.get("https://api.ipify.org")
print(driver.page_source)
driver.quit()

Note: Be sure to replace proxy_ip and port with actual values.

3. Using Proxies with Scrapy

Scrapy is a powerful framework for large-scale web scraping. Here's how to route its traffic through a proxy.

settings.py Configuration

In your Scrapy project, modify settings.py:

HTTP_PROXY = 'http://proxy_ip:port'

DOWNLOADER_MIDDLEWARES = {
    'scrapy.downloadermiddlewares.httpproxy.HttpProxyMiddleware': 110,
    'scrapy.downloadermiddlewares.useragent.UserAgentMiddleware': None,
}

Rotating Proxies in Scrapy

To rotate proxies, you can build a custom middleware or use a library like scrapy-rotating-proxies:

Install the library:

pip install scrapy-rotating-proxies

Then in settings.py:

ROTATING_PROXY_LIST = [
    'http://proxy1:port',
    'http://proxy2:port',
    'http://proxy3:port',
]

DOWNLOADER_MIDDLEWARES.update({
    'rotating_proxies.middlewares.RotatingProxyMiddleware': 610,
    'rotating_proxies.middlewares.BanDetectionMiddleware': 620,
})

Final Tips for Using Proxies in Python

1. Always test your proxies first using a simple request (like to https://api.ipify.org).
2. Rotate IPs if you're making many requests.
3. Set appropriate timeouts to avoid getting stuck on a dead proxy.
4. Respect target sites' robots.txt and rate limits.
5. Make sure your IP is whitelisted in the ProxiesThatWork dashboard.

With these configurations in place, you’ll be ready to integrate proxy support into almost any Python-based project.

Need help debugging? Check out our guide: Troubleshooting Proxy Connection Errors: A Step-by-Step Guide.