Synchronizing sqlite database from memory to file

Question

I\'m writing an application which must log information pretty frequently, say, twice in a second. I wish to save the information to an sqlite database, however I don\'t mind

Answer 1

Let's assume you have an on-disk database called 'disk_logs' with a table called 'events'. You could attach an in-memory database to your existing database:

ATTACH DATABASE ':memory:' AS mem_logs;

Create a table in that database (which would be entirely in-memory) to receive the incoming log events:

CREATE TABLE mem_logs.events(a, b, c);

Then transfer the data from the in-memory table to the on-disk table during application downtime:

INSERT INTO disk_logs.events SELECT * FROM mem_logs.events;

And then delete the contents of the existing in-memory table. Repeat.

This is pretty complicated though... If your records span multiple tables and are linked together with foreign keys, it might be a pain to keep these in sync as you copy from an in-memory tables to on-disk tables.

Before attempting something (uncomfortably over-engineered) like this, I'd also suggest trying to make SQLite go as fast as possible. SQLite should be able to easily handly > 50K record inserts per second. A few log entries twice a second should not cause significant slowdown.