What is the state of the art in HTML content extraction?

Question

There\'s a lot of scholarly work on HTML content extraction, e.g., Gupta & Kaiser (2005) Extracting Content from Accessible Web Pages, and some signs of interest here, e.g.,

Answer 1

One word: boilerpipe.

For the news domain, on a representative corpus, we're now at 98% / 99% extraction accuracy (avg/median)

• Demo: http://boilerpipe-web.appspot.com/
• Code: http://code.google.com/p/boilerpipe/
• Presentation: http://videolectures.net/wsdm2010_kohlschutter_bdu/
• Dataset and slides: http://www.l3s.de/~kohlschuetter/boilerplate/
• PhD thesis: http://www.kohlschutter.com/pdf/Dissertation-Kohlschuetter.pdf

Also quite language independent (today, I've learned it works for Nepali, too).

Disclaimer: I am the author of this work.