How can I asynchronously load data from large files in Qt?

Question

I\'m using Qt 5.2.1 to implement a program that reads in data from a file (could be a few bytes to a few GB) and visualises that data in a way that\'s dependent on every byt

Answer 1

For a hex viewer, I don't think you're on the right track at all - unless you think it will most likely be used on system with SCSI or RAID arrays for speed. Why load gigabytes of data at a time at all? A file access to fill up a text box happens pretty fast these days. Granted, that, e.g. Notepad++ has an excellent hex viewer plugin, and you have to load the file first; but that's because the file may be edited, and that's the way NPP works.

I think you would likely wind up subclassing a text box, going and getting enough data to load up the text box, or even splurge, and load up 500k of data before and after the current position. Then, say you are starting at byte zero. Load up enough data for your display, and maybe some extra data besides; but set the scrollbar type to always visible. Then, I think you'll probably intercept the scroll events by subclassing QTextBox; and writing your own scrollContentsBy() and changeEvent() and/or paint() event.

Even more simply, you could just create a QTextBox with no scrollbars ever; and a QVerticalScrollbar beside it. Set it's range and starting value. Then, respond to valueChanged() event; and change the contents of the QTextBox. That way, the user doesn't have to wait for along disk read in order to start editing, and it'll be a lot easier on resources (i.e. memory - so that if a lot of apps are open, they don't e.g. get swapped out to disk). It sounds hard to subclass these things, but a lot of times, it seems harder than it actually is. There are often fine examples already of somebody doing something like that.

If you have multiple threads reading a file, by contrast, you may have one reading from the beginning, another from the middle, and another towards the end. A single read head will be jumping around, trying to satisfy all requests, and therefore operate less efficiently. If it is an SDD drive instead, non-linear reads won't hurt you, but they won't help you either. If you'd prefer to make the trade-off of having a perhaps noticeable loading time, so that a user can scroll around a lot willy-nilly, a little faster (a textbox full of data to read really doesn't take very long to load, after all) then you might have a single thread read it in in the background, and then you can let the main one keep processing the event loop. More simply yet, just read in blocks of n megabytes at a time as it opens the whole file at once, and then do a qApp->processEvents(); to let the GUI respond to any GUI events that may have transpired in the meantime after every block read.

If you do believe it will most likely be used on a SCSI or RAID array, then it may make sense to do multitreading to do the reading. A SCSI drive can have multiple read heads; and some RAID arrays are set up to spread their data across multiple disks for speed purposes. Note that you would be better off using a single thread to do the reading, if the RAID array is set up to keep multiple identical copies of the data for data security purposes. When I went to implement multi-threading, I found the lightweight model proposed here most helpful: QThread: You were not doing so wrong. I did have to do Q_DECLARE_METATYPE on the result structure, have a constructor, destructor, and a move operator defined for it (I used memmove), and did a qRegisterMetaType() on both the structure, and the vector to hold the results, for it to return the results right. You pay the price of it blocking the vector in order to return its results; but the actually overhead of that didn't seem to be much at all. Shared memory might be worth pursuing in this context, too - but maybe each thread could have its own, so you won't need to lock out reads from other thread results to write it.