Scrappy response different than browser response

Question

I am trying to scrape a this page with scrapy:

http://www.barnesandnoble.com/s?dref=4815&sort=SA&startat=7391

and the response which I get is different than what I see in the browser. Browser response has the correct page, while scrapy response is:

http://www.barnesandnoble.com/s?dref=4815&sort=SA&startat=1

page. I have tried with urllib2 but still have the same issue. Any help is much appreciated.

Answer 1

I don't really understand the issue, but usually a different response for a browser and scrapy is caused by one these:

• the server analyzes your User-Agent header, and returns a specially crafted page for mobile clients or bots;
• the server analyzes the cookies, and does something special when it looks like you are visiting for the first time;
• you are trying to make a POST request via scrapy like the browser does, but you forgot some form fields, or put wrong values
• etc.

There is no universal way to determine what's wrong, because it depends on the server logic, which you don't know. If you are lucky, you will analyze and fix all the mentioned issues and will make it work.