Scrapy 404 error: HTTP status code is not handled or not allowed

Question

I\'m trying to scrape the site coursetalk using scrapy, I\'m trying with the spider template first and getting a 404 error:

2017-12-29 23:34:30 [scrapy] DEB         


        

Answer 1

I have faced this problem with scrapy and solved it.

Changed USER_AGENT in setting.py

USER_AGENT = "Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/34.0.1847.131 Safari/537.36"