Optimizing MySQL for ALTER TABLE of InnoDB

Question

Sometime soon we will need to make schema changes to our production database. We need to minimize downtime for this effort, however, the ALTER TABLE statements are going to

Answer 1

I tested various strategies to speed up one alter table. Eventually I got about 10x speed increase in my particular case. The results may or may not apply to your situation. However, based on this I would suggest experimenting with InnoDB log file/buffer size parameters.

In short, only increasing innodb_log_file_size and innodb_log_buffer_size had a measurable effect (Be careful! Changing innodb_log_file_size is risky. Look below for more info).

Based on the rough write data rate (iostat) and cpu activity the bottleneck was io based, but not data throughput. In the faster 500s runs the write throughput is at least in the same ballpark that you would expect from the hard disk.

Tried performance optimizations:

• select * into outfile + load data infile as suggested in https://stackoverflow.com/a/5303805/1148030
• "disable keys" I missed the fact that it has no effect in InnoDB which is true also based on the results. This was on in all cases except the final alter table. How to disable index in innodb
• innodb_buffer_pool_size, "Disk I/O Tips" at http://dev.mysql.com/doc/refman/5.1/en/innodb-tuning.html
• foreign_key_checks=0 et al, "Bulk data loading tips" at http://dev.mysql.com/doc/refman/5.0/en/innodb-tuning.html
• Bigger innodb_log_file_size (See warnings below) and innodb_log_buffer_size. See "Logging tips" at http://dev.mysql.com/doc/refman/5.0/en/innodb-tuning.html and http://dev.mysql.com/doc/refman/5.1/en/innodb-parameters.html#sysvar_innodb_log_file_size and http://dev.mysql.com/doc/refman/5.0/en/innodb-parameters.html#sysvar_innodb_log_buffer_size

Changing innodb_log_file_size can be dangerous. See http://www.mysqlperformanceblog.com/2011/07/09/how-to-change-innodb_log_file_size-safely/ The technique (file move) explained in the link worked nicely in my case.

Also see http://www.mysqlperformanceblog.com/2007/11/03/choosing-innodb_buffer_pool_size/ and http://www.mysqlperformanceblog.com/2008/11/21/how-to-calculate-a-good-innodb-log-file-size/ for information about innodb and tuning log sizes. One drawback of larger log files is longer recovery time after crash.

Test runs and rough timings:

• The simple load data to a freshly createad table: 6500s
• load data w. innodb_log_file_size=200M, innodb_log_buffer_size=8M, innodb_buffer_pool_size=2200M, autocommit= 0; unique_checks=0, foreign_key_checks=0: 500s
• load data w. innodb_log_file_size=200M, innodb_log_buffer_size=8M: 500s
• Equivalent straight alter table w. datainnodb_log_file_size=200M, innodb_log_buffer_size=8M: 500s

Testing details: Table: InnoDB, 6M rows, 2.8G on disk, single file (innodb_file_per_table option), primary key is 1 integer, +2 unque constraints/indices, 8 columns, avg. row length 218 bytes. Server: Ubuntu 12.04, x86_64, virtual machine, 8 cores, 16GB, sata consumer grade disk, no raid, no database activity, minuscule other process activity, minuscule activity in other and much smaller virtual machines. Mysql 5.1.53. The initial server config is pretty default except for increased innodb_buffer_pool_size of 1400M. The alter table adds 2 small columns. I didn't clock the raw alter table, but instead experimented with equivalent load data infile statement, finally I did the straight alter table and got comparable result.

This question is related to at least following questions:

• MySQL: add a field to a large table
• Add a new column in a large mysql table
• mysql slow query
• MySQL "set unique_checks", "set foreign_key_checks" vs. "alter table disable keys"
• Optimizing MySQL for ALTER TABLE of InnoDB