Qt widget for displaying large amount of data rows

Question

I am trying to display a large amount of columnar records in a scrollable view using Qt (5.1).

The number of rows I would like to be able to browse can vary from 100 million to 1 Billion, say.

The QTableWidget with a custom model works a few million rows, but the QTableWidget allocates data for each row because you can re-size the rows height, and so it must store data for this, which can use megabytes or even gigabytes of memory with 100M rows.

I do not require the re-sizeable rows functionality just a multi-column list would be ideal, but QTreeCtrl doesnt seem to work with many rows, and QList seems to only support single columns.

Should I be implementing a custom widget from QAbstractItemView for this purpose?

For those familiar with wxwidgets/wepython it can be done like this, and works well with billions of rows:

import wx

class VirtualList(wx.ListCtrl):

    def __init__(self, parent, id, pos, size, flags):
        wx.ListCtrl.__init__(self, parent, id, pos, size, flags)

    def OnGetItemText(self, item, column):
        return "Row %d, Column %d" % (item, column)

Answer 1

If it's tabular data I would use a table. I would write a custom QTableView with a custom QAbstractTableModel. In the QTableView you have control of all visible items. I would put some kind of check or variable shared between QTableView and it's model to control how much data should be shown. By overriding the data method in QAbstractTableModel you can dictate how much data to show. You can also mess with the QTableView's scroll bar to make things look and feel nicer.

If you don't really care about editing or looks, you could use a simple QTextEdit/QTextBrowser that is set to read only.

Note: A QTableWidget with a custom model is somewhat pointless. The main difference between a QTableWidget and a QTableView is that the QTableWidget has it's own premade model.

Answer 2

Qt views are known to be slow on large data amounts. QTableWidget or QStandardItemModel cannot be used here because they create new object for each table item. That causes large overhead. You should start from implementing your own fast QAbstractItemModel subclass and showing it in a standard QTableView. It's possible that it will work acceptably fast. You can also try to set fixed row height for the view to speed things up.

If it won't work, you can try to implement your own table view. Also you can use Graphics View Framework as described in this answer.