How to send JavaScript and Cookies Enabled in Scrapy?

Question

I am scraping a website using Scrapy which require cooking and java-script to be enabled. I don\'t think I will have to actually process javascript. All I need is to pretend as

Answer 1

AFAIK, there is no a universal solution. You have to debug the site, to see how it determines that Javascript is not supported/enabled by your client.

I don't think the server looks at X-JAVASCRIPT-ENABLED header. Maybe there is a cookie set by Javascript when the page loads in a real javascript enabled browser? Maybe the server looks at user-agent header?

See also this response.