Python scrape website w/BeautifulSoup4 shwoing attribute error for table with class name

问题

I am following this tutorial: https://www.pluralsight.com/guides/extracting-data-html-beautifulsoup

To download the table on this page: http://www.knapsackfamily.com/LunchBox/top.php#res

Edit: That table appears after I click the button "List All" which is an input in a form with action=top.php#res.

I inspected the table:

and it shows the table classes are either sortable dl or sortable d1 so I tried them both in my script:

"""
get knapsack food table and table at link "more"
follow: https://www.pluralsight.com/guides/extracting-data-html-beautifulsoup
"""

import pandas as pd
from bs4 import BeautifulSoup
import requests
import lxml.html as lh

pd.set_option('display.max_columns', None)
pd.set_option('display.max_rows', None)
url = "http://www.knapsackfamily.com/LunchBox/top.php#res"
#food_df = pd.read_csv(url)

#print(food_df)

page = requests.get(url).text
soup = BeautifulSoup(page, "lxml")
print(soup.prettify())

food_table = soup.find("table", attrs={"class": "sortable d1"})

food_table_data = food_table.tbody.find_all("tr")

headings=[]
# get all heading 
for th in food_table_data.find_all("th"):
    headings.append(th.b.text.replace('\n', ' ').strip())

print(headings)

but I get:

Traceback (most recent call last):
  File "get_knapsack_tables_to_csv.py", line 24, in <module>
    food_table_data = food_table.tbody.find_all("tr")
AttributeError: 'NoneType' object has no attribute 'tbody'

How can I fix this? I want to scrape it rather than using some methods in Pandas because I need to access the link the last column on that page called more and scrape that site's English language table cells and add them as columns to the dataframe I'm trying to make.

回答1:

To get data from server, use requests.post() with correct form data. For example:

import requests
from bs4 import BeautifulSoup

url = 'http://www.knapsackfamily.com/LunchBox/top.php'
data = {
    'mode': 'list3',
    'fword1': '',
    'mode1': ' List All'
}

soup = BeautifulSoup( requests.post(url, data=data).content, 'html.parser' )

for row in soup.select('table.d1 tr'):
    tds = [td.get_text(strip=True) for td in row.select('td, th')]
    print(*tds)

Prints:

大分類 KNApSAcK 種名 学名（一般名） 詳細データ
植物  Abelmoschus esculentus Abelmoschus esculentus[okra、おくら、オクラ、秋葵、あめりかねり、アメリカネリ、おかれんこん、オカレンコン、陸蓮根] more
植物  Abelmoschus moschatus Abelmoschus moschatus、Hibiscus abelmoschus[Ambrette、Musk seed、Musk mallow、とろろあおいもどき、トロロアオイモドキ、においとろろあおい、ニオイトロロアオイ、じゃこうあおい、ジャコウアオイ、りゅうきゅうとろろあおい、リュウキュウトロロアオイ] more
動物(魚類）  Abudefduf sexfasciatus Abudefduf sexfasciatus[scissor-tail sergeant、six-barred sergeant-major、ろくせんすずめだい、ロクセンスズメダイ、六線雀鯛] more
動物(魚類）  Abudefduf vaigiensis Abudefduf vaigiensis[five banded damsefish、sergeant major、waigieu damoiselle、おやびっちゃ、オヤビッチャ] more
植物  Acacia farnesiana Acacia farnesiana[Prickly Moses、Cassie、きんごうかん、キンゴウカン、金合歓] more
動物(魚類）  Acanthocepola krusensternii Acanthocepola krusensternii[yellowspotted bandfish、あかたち、アカタチ、赤太刀] more
動物(魚類）  Acanthocepola limbata Acanthocepola limbata[blackspot bandfish、いってんあかたち、イッテンアカタチ、一点赤太刀] more
動物(魚類）  Acanthocybium solandri Acanthocybium solandri[wahoo、かますさわら、カマスサワラ、魳鰆] more
動物(魚類）  Acanthogobius flavimanus Acanthogobius flavimanus[common blackish goby、genuine goby、spiny goby、まはぜ、マハゼ、真鯊、真沙魚、ごり、ゴリ、かじか、カジカ] more

...and so on.

来源：https://stackoverflow.com/questions/64636148/python-scrape-website-w-beautifulsoup4-shwoing-attribute-error-for-table-with-cl

标签

python

html

pandas

beautifulsoup