Download file using partial download (HTTP)

Question

Is there a way to download huge and still growing file over HTTP using the partial-download feature?

It seems that this code downloads file from scratch every time i

Answer 1

This is quite easy to do using TCP sockets and raw HTTP. The relevant request header is "Range".

An example request might look like:

mysock = connect(("www.example.com", 80))
mysock.write(
  "GET /huge-growing-file HTTP/1.1\r\n"+\
  "Host: www.example.com\r\n"+\
  "Range: bytes=XXXX-\r\n"+\
  "Connection: close\r\n\r\n")

Where XXXX represents the number of bytes you've already retrieved. Then you can read the response headers and any content from the server. If the server returns a header like:

Content-Length: 0

You know you've got the entire file.

If you want to be particularly nice as an HTTP client you can look into "Connection: keep-alive". Perhaps there is a python library that does everything I have described (perhaps even urllib2 does it!) but I'm not familiar with one.