I'm a Python newbie. I have been a little confused by the differences between the old urllib and urllib2 in Python 2.x and the new urllib in Python 3, and among other things I'm not sure when data needs to be encoded before being sent to urlopen.
I have been trying to fetch the html body of a url, using a POST so that I can send parameters. The webpage displays sunshine data for a country over a particular hour of a given day. I have tried without encoding/decoding and the printout is a string of bytes with b at the beginning. The code I then tried was
import urllib.request, urllib.parse, urllib.error def scrape(someurl): try: values = {'LANG': 'en', 'DATE' : '1303160400', 'CONT' : 'euro', 'LAND' : 'UK', 'KEY' : 'UK', 'SORT': '2', 'INT' : '06', 'TYPE' : 'sonnestd', 'ART' : 'karte', 'RUBRIK' : 'akt', 'R': '310', 'CEL': 'C'} data = urllib.parse.urlencode(values) data = data.encode("utf-8") response = urllib.request.urlopen(someurl, data) html = response.read().decode("utf-8") print(html) except urllib.error.HTTPError as e: print(e.code) print(e.read()) myscrape = scrape("http://www.weatheronline.co.uk/weather/maps/current") The error is
Traceback (most recent call last): File "/Users/Me/Desktop/weather.py", line 57, in <module> myscrape = scrape("http://www.weatheronline.co.uk/weather/maps/current") File "/Users/Me/Desktop/weather.py", line 37, in scrape html = response.read().decode("utf-8") UnicodeDecodeError: 'utf8' codec can't decode byte 0x80 in position 10: invalid start byte Without encoding/decoding I get a suspiciously short string of bytes anyway, so I wonder whether the request is failing in some other way
b'GIF89a\x01\x00\x01\x00\x80\x00\x00\x00\x00\x00\x00\x00\x00!\xf9\x04\x01\x00\x00\x00\x00,\x00\x00\x00\x00\x01\x00\x01\x00\x00\x02\x02D\x01\x00;'