BeautifulSoup does not work for some web sites

问题

I have this sript:

import urrlib2
from bs4 import BeautifulSoup
url = "http://www.shoptop.ru/"
page = urllib2.urlopen(url).read()
soup = BeautifulSoup(page)
divs = soup.findAll('a')
print divs

For this web site, it prints empty list? What can be problem? I am running on Ubuntu 12.04

回答1:

Actually there are quite couple of bugs in BeautifulSoup which might raise some unknown errors. I had a similar issue when working on apache using lxml parser

So, just try to use other couple of parsers mentioned in the documentation

soup = BeautifulSoup(page, "html.parser")

This should work!

回答2:

It looks like you have a few mistakes in your code urrlib2 should be urllib2, I've fixed the code for you and this works using BeautifulSoup 3

import urllib2
from BeautifulSoup import BeautifulSoup
url = "http://www.shoptop.ru/"
page = urllib2.urlopen(url).read()
soup = BeautifulSoup(page)
divs = soup.findAll('a')
print divs

来源：https://stackoverflow.com/questions/11650700/beautifulsoup-does-not-work-for-some-web-sites