Why can't I scrape Amazon by BeautifulSoup?

Question

Here is my python code:

import urllib2
from bs4 import BeautifulSoup

page = urllib2.urlopen("http://www.amazon.com/")
soup = BeautifulSoup(page)
pr         


        

Answer 1

I just ran into this and found that setting any user-agent will work. You don't need to lie about your user agent.

response = HTTParty.get @url, headers: {'User-Agent' => 'Httparty'}