How can I asynchronously load data from large files in Qt?

Question

I\'m using Qt 5.2.1 to implement a program that reads in data from a file (could be a few bytes to a few GB) and visualises that data in a way that\'s dependent on every byt

Answer 1

This seems like the case that you would want to have a consumer producer with semaphores. There is a very specific example which can walk you through properly implementing it. You need one more thread to make this work apart from your main thread.

The setup should be :

• Thread A runs your filereader as a producer
• You GUI thread runs your Hexviewer widget that consumes your data on specific events. Before issuing QSemaphore::acquire() a check with QSemaphore::available()` should be made in order to avoid blocking the GUI.
• Filereader and Hexviewer have access to a third class e.g. DataClass where the data is placed upon read and retrieved from the consumer. This should also have the semaphores defined.
• There is no need to emit a signal with the data or notify.

That pretty much covers moving your data read from filereader to your widget but it does not cover how to actually paint this data. In order to achive this you can consume the data within a paintevent by overriding the paint event of Hexviewer, and reading what has been put in the queue. A more elaborate approach would be to write an event filter.

On top of this you may want to have a maximum number of bytes read after which Hexviewer is explicitly signaled to consume the data.

Notice, that this solution is completely asynchronous, threadsafe and ordered, since none of your data is sent to Hexviewer, but the Hexviewer only consumes that when it needs to display on the screen.