Sqlite3: Disabling primary key index while inserting?

Question

I have an Sqlite3 database with a table and a primary key consisting of two integers, and I\'m trying to insert lots of data into it (ie. around 1GB or so)

The issue

Answer 1

1. You can't remove embedded index since it's the only address of row.
2. Merge your 2 integer keys in single long key = (key1<<32) + key2; and make this as a INTEGER PRIMARY KEY in youd schema (in that case you will have only 1 index)
3. Set page size for new DB at least 4096
4. Remove ANY additional index except primary
5. Fill in data in the SORTED order so that primary key is growing.
6. Reuse commands, don't create each time them from string
7. Set page cache size to as much memory as you have left (remember that cache size is in number of pages, but not number of bytes)
8. Commit every 50000 items.
9. If you have additional indexes - create them only AFTER ALL data is in table

If you'll be able to merge key (I think you're using 32bit, while sqlite using 64bit, so it's possible) and fill data in sorted order I bet you will fill in your first Gb with the same performance as second and both will be fast enough.

Answer 2

Are you doing the INSERT of each new as an individual Transaction?

If you use BEGIN TRANSACTION and INSERT rows in batches then I think the index will only get rebuilt at the end of each Transaction.

Answer 3

See faster-bulk-inserts-in-sqlite3.