Best practices for in-app database migration for Sqlite

Question

I am using sqlite for my iphone and I anticipate the database schema might change over time. What are the gotchas, naming conventions and things to watch out for to do a su

Answer 1

The answer from Just Curious is dead-on (you got my point!), and it's what we use to track the version of the database schema that is currently in the app.

To run through the migrations that need to occur to get user_version matching the app's expected schema version, we use a switch statement. Here's a cut-up example of what this look like in our app Strip:

- (void) migrateToSchemaFromVersion:(NSInteger)fromVersion toVersion:(NSInteger)toVersion { 
    // allow migrations to fall thru switch cases to do a complete run
    // start with current version + 1
    [self beginTransaction];
    switch (fromVersion + 1) {
        case 3:
            // change pin type to mode 'pin' for keyboard handling changes
            // removing types from previous schema
            sqlite3_exec(db, "DELETE FROM types;", NULL, NULL, NULL);
            NSLog(@"installing current types");
            [self loadInitialData];
        case 4:
            //adds support for recent view tracking
            sqlite3_exec(db, "ALTER TABLE entries ADD COLUMN touched_at TEXT;", NULL, NULL, NULL);
        case 5:
            {
                sqlite3_exec(db, "ALTER TABLE categories ADD COLUMN image TEXT;", NULL, NULL, NULL);
                sqlite3_exec(db, "ALTER TABLE categories ADD COLUMN entry_count INTEGER;", NULL, NULL, NULL);
                sqlite3_exec(db, "CREATE INDEX IF NOT EXISTS categories_id_idx ON categories(id);", NULL, NULL, NULL);
                sqlite3_exec(db, "CREATE INDEX IF NOT EXISTS categories_name_id ON categories(name);", NULL, NULL, NULL);
                sqlite3_exec(db, "CREATE INDEX IF NOT EXISTS entries_id_idx ON entries(id);", NULL, NULL, NULL);

               // etc...
            }
    }

    [self setSchemaVersion];
    [self endTransaction];
}

Answer 2

1. Create /migrations folder with the list of SQL-based migrations, where each migration looks something like this:

/migrations/001-categories.sql

-- Up
CREATE TABLE Category (id INTEGER PRIMARY KEY, name TEXT);
INSERT INTO Category (id, name) VALUES (1, 'Test');

-- Down
DROP TABLE User;

/migrations/002-posts.sql

-- Up
CREATE TABLE Post (id INTEGER PRIMARY KEY, categoryId INTEGER, text TEXT);

-- Down
DROP TABLE Post;

2. Create db table containing the list of applied migrations, for example:

CREATE TABLE Migration (name TEXT);

3. Update application bootstrap logic so that before it starts, it grabs the list of migrations from the /migrations folder and runs the migrations that have not yet been applied.

Here is an example implemented with JavaScript: SQLite Client for Node.js Apps

Answer 3

For .net you can use lib:

EntityFrameworkCore.Sqlite.Migrations

It is simple, so for any other platform you can easily implement the same behavior as in lib.

Answer 4

The best solution IMO is to build a SQLite upgrade framework. I had the same problem (in the C# world) and I built my own such framework. You can read about it here. It works perfectly and makes my (previously nightmarish) upgrades work with minimal effort on my side.

Although the library is implemented in C#, the ideas presented there should work fine in your case also.

Answer 5

If you change the database schema and all code that's using it in lockstep, as is likely to be the case in embedded and phone-located apps, the problem is actually well under control (nothing comparable to the nightmare that's schema migration on an enterprise DB that may be serving hundreds of apps -- not all under the DBA's control either;-).

Answer 6

I maintain an application that periodically needs to update a sqlite database and migrate old databases to the new schema and here's what I do:

For tracking the database version, I use the built in user-version variable that sqlite provides (sqlite does nothing with this variable, you are free to use it however you please). It starts at 0, and you can get/set this variable with the following sqlite statements:

> PRAGMA user_version;  
> PRAGMA user_version = 1;

When the app starts, I check the current user-version, apply any changes that are needed to bring the schema up to date, and then update the user-version. I wrap the updates in a transaction so that if anything goes wrong, the changes aren't committed.

For making schema changes, sqlite supports "ALTER TABLE" syntax for certain operations (renaming the table or adding a column). This is an easy way to update existing tables in-place. See the documentation here: http://www.sqlite.org/lang_altertable.html. For deleting columns or other changes that aren't supported by the "ALTER TABLE" syntax, I create a new table, migrate date into it, drop the old table, and rename the new table to the original name.