How to bypass cloudflare bot/ddos protection in Scrapy?

Question

I used to scrape e-commerce webpage occasionally to get product prices information. I have not used the scraper built using Scrapy in a while and yesterday was tryi

Answer 1

If you're getting 503 Error you can follow these guidelines:

1. Go to settings.py
2. Search for: USER_AGENT
3. Here you will see the default bot user agent by scrapy. Replace that default with this:

   USER_AGENT = 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/83.0.4103.116 Safari/537.36'