Data inserts take longer for more iterations executed

问题

I have an empty database which will contain a migrated form of an old legacy database.

I read in all of the old data into DataTables which works fine.

There is one master table which contains links for almost every table, so this is looped through. For every record that needs to go into the master table there are about 7 groups of tables, in each of which are only tables that rely on each other to work. so for example the Orders Table is in the same group as the OrderLine table as one relies on the other.

As each of these 7 groups can be done without any information from another group I start the migration process with a different thread for each group.

Each method simply runs through the relevant records from the legacy data table and sanitises them and inserts them into the new database.

I have a data access class that keeps an SQLCeConnection object open for the lifetime of the class.

Every insert and read operation hits these two methods:

/// <summary>
/// Executes a single INSERT, UPDATE, DELETE or other Sql Command that modifies the schema or data of the database
/// </summary>
/// <param name="sql">The command to execute</param>
/// <param name="parameters">Any parameters in the command</param>
public void ExecuteCommand(string sql, SqlServerCeParameter[] parameters)
{
    //print debug statements if necessary
    if (_outputSqlStatementsToFile == true) PrintSqlDebuggingInformation(sql, parameters);

    //create the command that will execute the Sql
    using (var command = new SqlCeCommand(sql, _connection))
    {
        //add any parameters
        if (parameters != null) command.Parameters.AddRange(parameters.Select(p => p.ParameterBehind).ToArray());

        //open the connection 
        if (_connection.State == ConnectionState.Closed)
        {
            _connection.Open();
        }

        //execute the command
        command.ExecuteNonQuery();

    }
}

 /// <summary>
    /// Executes a query that returns a single value, for example a COUNT(*) query
    /// </summary>
    /// <typeparam name="T">The type of the value returned by the query, for example COUNT(*) would be an Integer</typeparam>
    /// <param name="sql">The query to execute</param>
    /// <param name="parameters">Any parameters in the query</param>
    /// <returns>A single value cast to type T</returns>
    public T ExecuteQuery<T>(string sql, SqlServerCeParameter[] parameters)
    {
        //print debug statements if necessary
        if (_outputSqlStatementsToFile == true) PrintSqlDebuggingInformation(sql, parameters);

        //the result
        T result;

        //create the command that will execute the Sql
        using (var command = new SqlCeCommand(sql, _connection))
        {
            //add any parameters
            if (parameters != null) command.Parameters.AddRange(parameters.Select(p => p.ParameterBehind).ToArray());

            //open the connection 
            if (_connection.State == ConnectionState.Closed)
            {
                _connection.Open();
            }

            //execute the command
            var sqlResult = command.ExecuteScalar();

            //cast the result to the type given to the method
            result = (T)sqlResult;
        }
        //return the result
        return result;
    }

Every time one record is done that is the entire record and everything associated with that record fully migrated.

I have a stop watch running covering the entire code of the iteration so I can time the average of how many milliseconds per iteration it is taking.

At the beginning of the 32000+ rows the number of milliseconds is in the region of 180 - 220 milliseconds but as time goes on this figure steadily increases until it gets way over 2 seconds per iteration.

Each record is slightly different with some by nature taking longer to complete but I am pretty sure that there should not be this constant increase. I expected it to fluctuate widly in the early part of the migration then settle down to a relatively consistent figure.

I am wondering if it is something to do with the SQLServerCe connection, perhaps the more you use it without closing it the slower it gets?

1. C#
2. Visual Studio 2012
3. SqlServerCe 4.0

回答1:

You should consider taking a look at the clustered index on the target table. It should be small ( ideally and integer,) ascending, and unique. If you are using a business key for your clustered index or a guid, then you run the risk of page splits which would have the effect of slowing the load over time.

You may also consider dropping any foreign key constraints or indexes, then re-adding them upon completion.

This seems like something to do with indexes. An easy test to determine this is truncate the tables every 10K iterations or so. If you no longer see the slow-down, then it is likely due to the IO of inserting individual records.

来源：https://stackoverflow.com/questions/25914143/data-inserts-take-longer-for-more-iterations-executed