Spoofing IP address when web scraping (python)

Question

I have made a web scraper in python to give me information on when free bet offers from various bookie websites have changed or new ones have been added.

<

Answer 1

In order to overcome IP rate ban and hide your real IP you need to use proxies. There are a lot of different services that provide proxies. Consider using them as managing proxies by yourself is a real headache and cost would be much higher. I suggest https://botproxy.net among others. They provide rotating proxies though a single endpoint. Here is how you can make requests using this service:

#!/usr/bin/env python
import urllib.request
opener = urllib.request.build_opener(
    urllib.request.ProxyHandler(
        {'http': 'http://user-key:key-password@x.botproxy.net:8080',
         'https': 'http://user-key:key-password@x.botproxy.net:8080'}))
print(opener.open('https://httpbin.org/ip').read())

or using requests library

import requests

res = requests.get(
    'http://httpbin.org/ip',
    proxies={
        'http': 'http://user-key:key-password@x.botproxy.net:8080',
        'https': 'http://user-key:key-password@x.botproxy.net:8080'
        },
    headers={
        'X-BOTPROXY-COUNTRY': 'US'
        })
print(res.text)

They also have proxies in different countries.