Database file is inexplicably locked during SQLite commit

Question

I\'m performing a large number of INSERTS to a SQLite database. I\'m using just one thread. I batch the writes to improve performance and have a bit of security in case of

Answer 1

Things to watch for:

• don't use connections across multiple threads/processes.

• I've seen it happen when a virus scanner would detect changes to the file and try to scan it. It would lock the file for a short interval and cause havoc.

Answer 2

It looks like you failed to link the command with the transaction you've created. Instead of:

using (SQLiteCommand command = conn.CreateCommand())

You should use:

using (SQLiteCommand command = new SQLiteCommand("<INSERT statement here>", conn, trans))

Or you can set its Transaction property after its construction.

While we are at it - your handling of failures is incorrect:

The command's ExecuteNonQuery method can also fail and you are not really protected. You should change the code to something like:

   public void Commit()
    {
        using (SQLiteConnection conn = new SQLiteConnection(this.connString))
        {
            conn.Open();
            SQLiteTransaction trans = conn.BeginTransaction();
            try
            {
                using (SQLiteCommand command = conn.CreateCommand())
                {
                    command.Transaction = trans; // Now the command is linked to the transaction and don't try to create a new one (which is probably why your database gets locked)
                    command.CommandText = "INSERT OR IGNORE INTO [MY_TABLE] (col1, col2) VALUES (?,?)";

                    command.Parameters.Add(this.col1Param);
                    command.Parameters.Add(this.col2Param);

                    foreach (Data o in this.dataTemp)
                    {
                        this.col1Param.Value = o.Col1Prop;
                        this. col2Param.Value = o.Col2Prop;

                        command.ExecuteNonQuery();
                    }
                }

                trans.Commit();
            }
            catch (SQLiteException ex)
            {
                // You need to rollback in case something wrong happened in command.ExecuteNonQuery() ...
                trans.Rollback();
                throw;
            }
        }
    }

Another thing is that you don't need to cache anything in memory. You can depend on SQLite journaling mechanism for storing incomplete transaction state.

Answer 3

Run Sysinternals Process Monitor and filter on filename while running your program to rule out if any other process does anything to it and to see what exacly your program is doing to the file. Long shot, but might give a clue.

Answer 4

Is your database file on the same machine as the app or is it stored on a server?

You should create a new connection in every thread. I would simplefy the creation of a connection, use everywhere: connection = new SQLiteConnection(connString.ToString());

and use a database file on the same machine as the app and test again.

Why the two different ways of creating a connection?

Answer 5

We had a very similar problem using nested Transactions with the TransactionScope class. We thought all database actions occurred on the same thread...however we were caught out by the Transaction mechanism...more specifically the Ambient transaction.

Basically there was a transaction higher up the chain which, by the magic of ado, the connection automatically enlisted in. The result was that, even though we thought we were writing to the database on a single thread, the write didn't really happen until the topmost transaction was committed. At this 'indeterminate' point the database was written to causing it to be locked outside of our control.

The solution was to ensure that the sqlite database did not directly take part in the ambient transaction by ensuring we used something like:

using(TransactionScope scope = new TransactionScope(TransactionScopeOptions.RequiresNew))
{
  ...
  scope.Complete()
}

Answer 6

Do you have Google Desktop Search (or another file indexer) running? As previously mentioned, Sysinternals Process Monitor can help you track it down.

Also, what is the filename of the database? From PerformanceTuningWindows:

Be VERY, VERY careful what you name your database, especially the extension

For example, if you give all your databases the extension .sdb (SQLite Database, nice name hey? I thought so when I choose it anyway...) you discover that the SDB extension is already associated with APPFIX PACKAGES.

Now, here is the cute part, APPFIX is an executable/package that Windows XP recognizes, and it will, (emphasis mine) ADD THE DATABASE TO THE SYSTEM RESTORE FUNCTIONALITY

This means, stay with me here, every time you write ANYTHING to the database, the Windows XP system thinks a bloody executable has changed and copies your ENTIRE 800 meg database to the system restore directory....

I recommend something like DB or DAT.