Can SQLite handle 90 million records?

Question

Or should I use a different hammer to fix this problem.

I\'ve got a very simple use-case for storing data, effectively a sparse matrix, which I\'ve attempted to stor

Answer 1

Consider using a table for new inserts of the given day, without an index. Then, at the end of each day, run a script which will:

1. Insert new values from new_table into master_table
2. Clear the new_table for next day of processing

If you can do lookups on historical data in O(log n), and lookups on today's data in O(n), this should provide a nice compromise.