Beautiful soup just extract header of a table

问题

I want to extract information from the table in the following website using beautiful soup in python 3.5.

http://www.askapatient.com/viewrating.asp?drug=19839&name=ZOLOFT

I have to save the web-page first, since my program needs to work off-line.

I saved the web-page in my computer and I used the following codes to extract table information. But the problem is that the code just extract heading of the table.

This is my code:

from urllib.request import Request, urlopen
from bs4 import BeautifulSoup
url = "file:///Users/MD/Desktop/ZoloftPage01.html"


home_page= urlopen(url)
soup = BeautifulSoup(home_page, "html.parser")
table = soup.find("table", attrs={"class":"ratingsTable" } )
comments = [td.get_text() for td in table.findAll("td")]
print(comments)

And this is the output of the code:

['RATING', '\xa0 REASON', 'SIDE EFFECTS FOR ZOLOFT', 'COMMENTS', 'SEX', 'AGE', 'DURATION/DOSAGE', 'DATE ADDED ', '\xa0’]

I need all the information in the table’s rows. Thanks for your help !

回答1:

This is because of the broken HTML of the page. You need to switch to a more lenient parser like html5lib. Here is what works for me:

from pprint import pprint

import requests
from bs4 import BeautifulSoup

url = "http://www.askapatient.com/viewrating.asp?drug=19839&name=ZOLOFT"
response = requests.get(url, headers={'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_4) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/51.0.2704.103 Safari/537.36'})

# HTML parsing part
soup = BeautifulSoup(response.content, "html5lib")
table = soup.find("table", attrs={"class":"ratingsTable"})
comments = [[td.get_text() for td in row.find_all("td")] 
            for row in table.find_all("tr")]
pprint(comments)

来源：https://stackoverflow.com/questions/38680057/beautiful-soup-just-extract-header-of-a-table