How to Use a Proxy in Python requests

A proxy in Python requests functions as a gateway URL stored within a Session. When making a request, the call goes through this gateway first, causing the website to see the exit IP instead of the machine's own IP. The keys in the dict (proxies) denote the scheme of the URL being called, necessitating the gateway string for both http and https.

An empty https key sends the call out from the machine's address, with the echo service displaying your office IP. The proxies dict maps a target scheme to a gateway URL, with the key being http or https of the webpage being opened, not the type of proxy. A SOCKS gateway falls under the https key when the target begins with https://.

If the https entry is missing, requests sends the store, search page, or echo service directly, meaning the gateway's gigabytes never transfer. To enable SOCKS support in requests, install PySocks with pip install requests[socks]. A gateway URL starting with socks5h:// utilizes the gateway to resolve the hostname, with both the lookup and the exit IP appearing on the same hop, making the name and address part of the same journey.

An HTTP gateway uses a URL starting with http://. Sessions, which hold the gateway, headers, and cookie jar, are crucial objects that store this information. In subsequent calls within the same run, later requests reuse the jar, ensuring the first response's exit IP continues to the next page. To set up a Session, import os, urllib.parse, and requests, then create a Session object, disable trust_env to prevent a leftover HTTPS_PROXY from overriding your new gateway, and set the proxies dict to the gateway URL for both http and https.

Use quote from urllib.parse to safely encode any password containing @ or : inside the userinfo. Once a proxy is set, the Session will print the real IP if the https key is absent or if an HTTPS_PROXY in the shell has replaced the gateway. If the gateway does not respond, a timeout ends the wait instead of leaving the process frozen.

After the printed address matches your desired country, point the same Session at the real host. The username in the gateway corresponds to the country and session mode. Rotating users provide a new exit IP for each request, while sticky users maintain one exit IP for a single job before releasing it. For product lists requiring multiple exit IPs, rotating users are beneficial.

The host, port, username, and password originate from the provider dashboard. IPOasis offers dynamic residential proxies at $0.78 per GB, supporting both HTTP and SOCKS5 on the same balance, with unused gigabytes remaining on the balance. In the FAQ, it explains how to set a proxy in Python requests by placing a gateway URL on Session.proxies for both http and https.

To set a proxy, provide a gateway URL on Session.proxies for both http and https. The Session sends each call through this gateway, displaying the exit IP. A single requests.get with a proxies argument works for individual calls, while cookie jar information resets with the next call. socks5h refers to a SOCKS5 URL where the hostname is resolved at the gateway, causing both the DNS lookup and the exit IP to occur on the same hop.

The extra package required is requests[socks]. An HTTP gateway uses an http:// URL, and the dashboard indicates which format you have. The Session prints the real IP when the https key is absent or when HTTPS_PROXY in the shell has replaced the gateway. requests chooses the dict entry based on the target's scheme, meaning an https:// echo request bypasses an http-only map.

Trusting the environment prevents leftover HTTPS_PROXY values from replacing the newly set gateway. Both keys require the same gateway string, so trust_env = False is employed. A sticky session keeps one exit IP for a single job before releasing it. Once the checkout is complete, the next request can use a different household IP.

The dashboard generates the username, which is stored in PROXY_USER. The example reads PROXY_USER, PROXY_PASS, and PROXY_HOST from environment variables. The host value includes the port from the dashboard. Keeping the password out of the source file ensures a pasted snippet doesn't contain a live login, and quote encodes @ and : before constructing the URL.