How to download chunked data with Pythons urllib2

Question

I\'m trying to download a large file from a server with Python 2:

req = urllib2.Request(\"https://myserver/mylargefile.gz\")
rsp = urllib2.urlopen(req)
data          


        

Answer 1

I have the same problem.

I found that "Transfer-Encoding: chunked" often appears with "Content-Encoding: gzip".

So maybe we can get the compressed content and unzip it.

It works for me.

import urllib2
from StringIO import StringIO
import gzip

req = urllib2.Request(url)
req.add_header('Accept-encoding', 'gzip, deflate')
rsp = urllib2.urlopen(req)
if rsp.info().get('Content-Encoding') == 'gzip':
    buf = StringIO(rsp.read())
    f = gzip.GzipFile(fileobj=buf)
    data = f.read()