Web scraping results in 403 Forbidden Error

Question

I\'m trying to web scrape the earnings for each company off SeekingAlpha using BeautifulSoup. However, it seems like the site is detecting that a web scraper is being used?

Answer 1

For anyone out there using PyQuery:

from pyquery import PyQuery as pq
import requests


page = pq('https://seekingalpha.com/article/4151372-tesla-fools-media-model-s-model-x-demand', proxies={'http':'34.231.147.235:8080'})
print(page)

• (Used proxy info from https://free-proxy-list.net/)
• Make sure you are using the Requests library and not Urllib. Don't try and load a page with 'urlopen'.