retrieving just the title of a webpage in python

问题

I have more than 5000 webpages i want the titles of all of them. In my project i am using the BeautifulSoup html parser like this.

soup = BeautifulSoup(open(url).read())
soup('title')[0].string

But its taking lots of time. Just for the title of a webpage i am reading the entire file and building the parse tree(I thought this is the reason for delay, correct me if i am wrong).

Is there in any other simple way to do this in python.

回答1:

It would certainly be faster if you just used a simple regular expression, BeautifulSoup is pretty slow. You could do something like:

import re
regex = re.compile('<title>(.*?)</title>', re.IGNORECASE|re.DOTALL)
regex.search(string_to_search).group(1)

回答2:

You could always use a regular expression to do it, but that might break if you get a badly formatted page. That'd be something like this:

import re
titleRE = re.compile("<title>(.+?)</title>")
title = titleRE.search(s).group(1)

回答3:

You could even use the simple string methods:

html = '<html> lots of crap <title>Title</title> even more crap </html>'
start = html.find('<title>') + 7 # Add length of <title> tag
end = html.find('</title>', start)
title = html[start:end]

However, this only guarantees that <title> is found before </title> in the page. Not that it is in the <head> section or anything.

Also, you should validate your assumption that it actually is the BeautifulSoup parsing that comsumes the lion's share of the time. (My guess is that open(url).read() for 5,000 resources takes quite some time, too. This you will not eliminate, no matter how you "parse" the HTML.)

回答4:

Try

>> hearders = {'headers':'Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:51.0) Gecko/20100101 Firefox/51.0'}
>>> n = requests.get('http://www.imdb.com/title/tt0108778/', headers=hearders)
>>> al = n.text
>>> al[al.find('<title>') + 7 : al.find('</title>')]
u'Friends (TV Series 1994\u20132004) - IMDb'

来源：https://stackoverflow.com/questions/5960087/retrieving-just-the-title-of-a-webpage-in-python

标签

python

html

beautifulsoup