Android SQLite and huge data sets

Question

We are creating an app for a client that has hundreds of megabytes of HTML in SQLite databases. We have implemented a way to query this data and scroll through it all in a r

Answer 1

In my experience, limiting queries makes it take a lot longer to get results because starting new cursors is expensive for low limits. I tried doing 62k rows with 1k and even 10k limit just now and it is very slow and unusable since I have to start more than 6 cursors. I'll just stick with not supporting 2.3.3.... It is the latest build that I get CursorWindow ERROR instead of WARN.

Here is what I did though. This is probably the best algorithm I think. Don't have to do queries twice etc. However, it can get pretty slow with big queries and small limits so you have to test out what works best a lot. In my case, it's not fast enough for my purposes since it doesn't handle 62k rows well.

int cursorCount = 0;
int limit = 1000; //whatever you want
while (true)
{
    Cursor cursor = builder.query(mDatabaseHelper.getReadableDatabase(), columns, selection, selectionArgs, null, null, null, Integer.toString(limit));
    if (cursor == null) {
        return null;
    } else if (!cursor.moveToFirst()) { //if it is empty, return null
        return null;
    }
    cursorCount = cursor.getCount();
    if (cursorCount % limit != 0)
    {
        return cursor;
    }
    limit+=1000; //same amount as the one above
}