Update MySQL table in chunks

问题

I am trying to update a MySQL InnoDB table with c. 100 million rows. The query takes close to an hour, which is not a problem.

However, I'd like to split this update into smaller chunks in order not to block table access. This update does not have to be an isolated transaction.

At the same time, the splitting of the update should not be too expensive in terms of additional overhead.

I considered looping through the table in a procedure using :

UPDATE TABLENAME SET NEWVAR=<expression> LIMIT batchsize, offset,

But UPDATE does not have an offset option in MySQL.

I understand I could try to UPDATE ranges of data that are SELECTed on a key, together with the LIMIT option, but that seems rather complicated for that simple task.

回答1:

I ended up with the procedure listed below. It works but I am not sure whether it is efficient with all the queries to identify consecutive ranges. It can be called with the following arguments (example):

call chunkUpdate('SET var=0','someTable','theKey',500000);

Basically, the first argument is the update command (e.g. something like "set x = ..."), followed by the mysql table name, followed by a numeric (integer) key that has to be unique, followed by the size of the chunks to be processed. The key should have an index for reasonable performance. The "n" variable and the "select" statements in the code below can be removed and are only for debugging.

delimiter //
CREATE PROCEDURE chunkUpdate (IN cmd VARCHAR(255), IN tab VARCHAR(255), IN ky VARCHAR(255),IN sz INT)
BEGIN
  SET @sqlgetmin = CONCAT("SELECT MIN(",ky,")-1 INTO @minkey FROM ",tab); 
  SET @sqlgetmax = CONCAT("SELECT MAX(",ky,") INTO @maxkey FROM ( SELECT ",ky," FROM ",tab," WHERE ",ky,">@minkey ORDER BY ",ky," LIMIT ",sz,") AS TMP"); 
  SET @sqlstatement = CONCAT("UPDATE ",tab," ",cmd," WHERE ",ky,">@minkey AND ",ky,"<=@maxkey");
  SET @n=1;

  PREPARE getmin from @sqlgetmin;
  PREPARE getmax from @sqlgetmax;
  PREPARE statement from @sqlstatement;

  EXECUTE getmin;

  REPEAT
    EXECUTE getmax; 
    SELECT cmd,@n AS step, @minkey AS min, @maxkey AS max;
    EXECUTE statement;
    set @minkey=@maxkey;
    set @n=@n+1;
  UNTIL @maxkey IS NULL
  END REPEAT; 
  select CONCAT(cmd, " EXECUTED IN ",@n," STEPS") AS MESSAGE;
END//

来源：https://stackoverflow.com/questions/27938958/update-mysql-table-in-chunks

标签

mysql

locking

sql-update