How to download any(!) webpage with correct charset in python?

Question

Problem

When screen-scraping a webpage using python one has to know the character encoding of the page. If you get the character encoding wrong th

Answer 1

Use the Universal Encoding Detector:

>>> import chardet
>>> chardet.detect(urlread("http://google.cn/"))
{'encoding': 'GB2312', 'confidence': 0.99}

The other option would be to just use wget:

  import os
  h = os.popen('wget -q -O foo1.txt http://foo.html')
  h.close()
  s = open('foo1.txt').read()