Best way to archive live MySQL database

Question

We have a live MySQL database that is 99% INSERTs, around 100 per second. We want to archive the data each day so that we can run queries on it without affecting the main, l

Answer 1

http://www.maatkit.org/ has mk-archiver

archives or purges rows from a table to another table and/or a file. It is designed to efficiently “nibble” data in very small chunks without interfering with critical online transaction processing (OLTP) queries. It accomplishes this with a non-backtracking query plan that keeps its place in the table from query to query, so each subsequent query does very little work to find more archivable rows.

Another alternative is to simply create a new database table each day. MyIsam does have some advantages for this, since INSERTs to the end of the table don't generally block anyway, and there is a merge table type to being them all back together. A number of websites log the httpd traffic to tables like that.

With Mysql 5.1, there are also partition tables that can do much the same.

Answer 2

MK-ARCHIVER is a elegant tool to archive MYSQL data.

http://www.maatkit.org/doc/mk-archiver.html

Answer 3

MySQL replication would work perfectly for this.
Master -> the live server.
Slave -> a different server on the same network.

Answer 4

I use mysql partition tables and I've achieve wonderful results in all aspects.

Answer 5

Could you keep two mirrored databases around? Write to one, keep the second as an archive. Switch every, say, 24 hours (or however long you deem appropriate). Into the database that was the archive, insert all of todays activity. Then the two databases should match. Use this as the new live db. Take the archived database and do whatever you want to it. You can backup/extract/read all you want now that its not being actively written to.

Its kind of like having mirrored raid where you can take one drive offline for backup, resync it, then take the other drive out for backup.

Answer 6

Sounds like replication is the best solution for this. After the initial sync the slave gets updates via the Binary Log, thus not affecting the master DB at all.

More on replication.