问题
I'd like to build a webapp to help other students at my university create their schedules. To do that I need to crawl the master schedules (one huge html page) as well as a link to a detailed description for each course into a database, preferably in python. Also, I need to log in to access the data.
- How would that work?
- What tools/libraries can/should I use?
- Are there good tutorials on that?
- How do I best deal with binary data (e.g. pretty pdf)?
- Are there already good solutions for that?
回答1:
- requests for downloading the pages.
- Here's an example of how to login to a website and download pages: https://stackoverflow.com/a/8316989/311220
- lxml for scraping the data.
If you want to use a powerful scraping framework there's Scrapy. It has some good documentation too. It may be a little overkill depending on your task though.
回答2:
Scrapy is probably the best Python library for crawling. It can maintain state for authenticated sessions.
Dealing with binary data should be handled separately. For each file type, you'll have to handle it differently according to your own logic. For almost any kind of format, you'll probably be able to find a library. For instance take a look at PyPDF for handling PDFs. For excel files you can try xlrd.
回答3:
I liked using BeatifulSoup for extracting html data
It's as easy as this:
from BeautifulSoup import BeautifulSoup
import urllib
ur = urllib.urlopen("http://pragprog.com/podcasts/feed.rss")
soup = BeautifulSoup(ur.read())
items = soup.findAll('item')
urls = [item.enclosure['url'] for item in items]
回答4:
For this purpose there is a very useful tool called web-harvest Link to their website http://web-harvest.sourceforge.net/ I use this to crawl webpages
来源：https://stackoverflow.com/questions/8335630/how-to-crawl-a-website-extract-data-into-database-with-python