Basic fetching of a URL's HTML body with Python 3.x

Question

I'm a Python newbie. I have been a little confused by the differences between the old urllib and urllib2 in Python 2.x and the new urllib in Python 3, and among other things I'm not sure when data needs to be encoded before being sent to urlopen.

I have been trying to fetch the html body of a url, using a POST so that I can send parameters. The webpage displays sunshine data for a country over a particular hour of a given day. I have tried without encoding/decoding and the printout is a string of bytes with b at the beginning. The code I then tried was

import urllib.request, urllib.parse, urllib.error

def scrape(someurl):

    try:

        values = {'LANG': 'en',
                  'DATE' : '1303160400',
                  'CONT' : 'euro',
                  'LAND' : 'UK',
                  'KEY' : 'UK',
                  'SORT': '2',
                  'INT' : '06',
                  'TYPE' : 'sonnestd',
                  'ART' : 'karte',
                  'RUBRIK' : 'akt',
                  'R': '310',
                  'CEL': 'C'}

        data = urllib.parse.urlencode(values)
        data = data.encode("utf-8")
        response = urllib.request.urlopen(someurl, data)
        html = response.read().decode("utf-8")
        print(html)

    except urllib.error.HTTPError as e:
        print(e.code)
        print(e.read())

myscrape = scrape("http://www.weatheronline.co.uk/weather/maps/current")

The error is

Traceback (most recent call last):
  File "/Users/Me/Desktop/weather.py", line 57, in <module>
    myscrape = scrape("http://www.weatheronline.co.uk/weather/maps/current")
  File "/Users/Me/Desktop/weather.py", line 37, in scrape
    html = response.read().decode("utf-8")
UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 10: invalid start byte

Without encoding/decoding I get a suspiciously short string of bytes anyway, so I wonder whether the request is failing in some other way

b'GIF89a\x01\x00\x01\x00\x80\x00\x00\x00\x00\x00\x00\x00\x00!\xf9\x04\x01\x00\x00\x00\x00,\x00\x00\x00\x00\x01\x00\x01\x00\x00\x02\x02D\x01\x00;'

Answer 1

The GIF89a means that the server is sending you an image.

Also, you shouldn't be brute-force decoding with UTF-8 anyway; you should look at the response headers to figure out which encoding to use.