Read and save part of file stream via IdHTTP

Question

I want to download a file from a HTTP server via a file stream and only read (and save to a file) the first few lines, say 100. After reading the first 100 lines the file stream must end: so I do NOT want to download or read the entire file.

Below you can find what I have so far. The website is just an example. Can someone guide me in the right direction?

const
  myURL = https://graphical.weather.gov/xml/DWMLgen/schema/latest_DWML.txt
var
  fs: TMemoryStream;
  http: TIdHTTP; 
begin
  fs := TMemoryStream.Create;
  http := TIdHTTP.Create(nil);
  try
    fs.Position := 0;
    http.Get(myURL, fs);
    fs.SaveToFile('test.xml');
  finally
    fs.Free;
    http.free
  end;
end;

Answer 1

If the HTTP server supports byte ranges for the desired URL (via the Range request header), you can request just the specific bytes you want, and that is all the server will send. You can use the TIdHTTP.Request.Range property for that when calling TIdHTTP.Get(). To discover if the server supports byte ranges, use TIdHTTP.Head() first to get the URL's headers, and then check if there is an Accept-Ranges: bytes header present (see the TIdHTTP.Response.AcceptRanges property).

If the server does not support byte ranges, you will have to continue using the code you currently have, just make some changes to it:

• Instead of calling fs.SaveToFile(), create a separate TFileStream object and pass the TMemoryStream to its CopyFrom() method, that way you can specify exactly how many bytes to save.

• Use the TIdHTTP.OnWork event, or use a TIdEventStream, or derive a custom TStream that overrides Write(), in order to keep track of how many bytes are being downloaded, so you can abort the download (by raising an exception, like EAbort via SysUtils.Abort()) once the desired number of bytes have been received.

Of course, either approach is byte oriented not line oriented. If you need to be line-oriented, and particularly if the lines are variable length, then you will have to use the second approach above, using TIdEventStream or a custom TStream, so you can implement line parsing logic and save only complete lines to your file, and then abort once you have received the desired number of lines.