Identifying large bodies of text via BeautifulSoup or other python based extractors

Question

Given some random news article, I want to write a web crawler to find the largest body of text present, and extract it. The intention is to extract the physical news article on the page.

The original plan was to use a BeautifulSoup findAll(True) and to sort each tag by its .getText() value. EDIT: don't use this for html work, use the lxml library, it's python based and much faster than BeautifulSoup. command (which means extract all html tags)

But this won't work for most pages, like the one I listed as an example, because the large body of text is split into many smaller tags, like paragraph dividers for example.

Does anyone have any experience with this? Any help with something like this would be amazing.

At the moment I'm using BeautifulSoup along with python, but willing to explore other possibilities.

---

EDIT: Came back to this question after a few months later (wow i sounded like an idiot ^), and solved this with a combination of libraries & own code.

Here are some deadly helpful python libraries for the task in sorted order of how much it helped me:

1 goose library Fast, powerful, consistent

2 readability library Content is passable, slower on average than goose but faster than boilerpipe

3 python-boilerpipe Slower & hard to install, no fault to the boilerpipe library (originally in java), but to the fact that this library is build on top of another library in java, which attributes to IO time & errors, etc.

I'll release benchmarks perhaps if there is interest.

---

Indirectly related libraries, you should probably install them and read their docs:

• NLTK text processing library This is too good not to install. They provide text analysis tools along with html tools (like cleanup, etc).
• lxml html/xml parser Mentioned above. This beats BeautifulSoup in every aspect but usability. It's a bit harder to learn but the results are worth it. HTML parsing takes much less time, it's very noticeable.
• python webscrapper library I think the value of this code isn't the lib itself, but using the lib as a reference manual to build your own crawlers/extractors. It's very nicely coded / documented!

A lot of the value and power in using python, a rather slow language, comes from it's open source libraries. They are especially awesome when combined and used together, and everyone should take advantage of them to solve whatever problems they may have!

Goose library gets lots of solid maintenance, they just added Arabic support, it's great!

Answer 1

You might look at the python-readability package which does exactly this for you.

Answer 2

You're really not going about it the right way, I would say, as all the comments above would attest to.

That said, this does what you're looking for.

from bs4 import BeautifulSoup as BS
import requests
html = requests.get('http://www.cnn.com/2013/01/04/justice/ohio-rape-online-video/index.html?hpt=hp_c2').text
soup = BS(html)
print '\n\n'.join([k.text for k in soup.find(class_='cnn_strycntntlft').find_all('p')])

It pulls out only the text, first by finding the main container of all the <p> tags, then by selecting only the <p> tags themselves to get the text; ignoring the <script> and other irrelevant ones.

As was mentioned in the comments, this will only work for CNN--and possibly, only this page. You might need a different strategy for every new webpage.