Web scraping results in 403 Forbidden Error

Question

I\'m trying to web scrape the earnings for each company off SeekingAlpha using BeautifulSoup. However, it seems like the site is detecting that a web scraper is being used?

Answer 1

You should try setting User-Agent as one of request headers. Value can be of any known browser.

Example:

Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_6) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36