The de facto real world HTML parser in Python is beautiful soup. The Python requests library is popular these days for HTTP (although the standard library has similar functionality but with a rather cumbersome API).
The scrappy and harvestman projects are real world crawlers that have been custom built just for the purpose of crawling.
