Deleting millions of rows in MySQL

Question

I recently found and fixed a bug in a site I was working on that resulted in millions of duplicate rows of data in a table that will be quite large even without them (still

Answer 1

I have faced similar issue while deleting multiple records from transaction table after moving them to archival table.

I used to use temporary table to identify records to be deleted.

The temporary table that I used 'archive_temp' to store ids created in memory without any indexes.

Hence while deleting records from original transaction table as e.g. DELETE from tat where id in (select id from archive_temp); query used to return an error "LOST Connection to server"

I created index on that temporary table as follows after creating it: ALTER TABLE archive_temp ADD INDEX( id);

After this my delete query used to execute in less than seconds irrespective of number of records to be deleted from transaction table.

Hence it would be better to check indexes. Hope this might help.