getting Forbidden by robots.txt: scrapy

Question

while crawling website like https://www.netflix.com, getting Forbidden by robots.txt: https://www.netflix.com/>

ERROR: No response downloaded for: https://www.netfli

Answer 1

First thing you need to ensure is that you change your user agent in the request, otherwise default user agent will be blocked for sure.