MySQLi : Inserting multiple rows with one prepared statement

Question

I made a script that creates a raw query string and then inserts the hundreds of rows in one statement. It works but does not offer the protections that prepared statements do. I then modified my script to add prepared statements. It works however, it is significantly slower. The script with prepared statements takes much longer to insert the rows than the raw query script, due to the script running through each prepared insert statement one row at a time rather than inserting hundred of rows at a time.

Here's a snippet of the prepared statement code:

for( $j = 0; $j < $abilitiesMax - 2; $j++ ){
  $stmtAbility->bind_param('iiiii', $abilityArray[$i]["match_id"] , $abilityArray[$i]["player_slot"],
  $abilityArray[$i][$j]["ability"], $abilityArray[$i][$j]["time"], $abilityArray[$i][$j]["level"] );

  if(  !($stmtAbility->execute()) ){      
   echo "<p>$db->error</p>";
   echo "<p>ERROR: when trying to insert abilities query</p>";
  }
}

It gets the job done but only after hundreds upon hundreds of inserts. Is there a way to bind lists or arrays to the bind_param() arguments and just run the $stmtAbility->execute one time or some other method that can speed up performance.

Sorry if this has been asked and answered before. I looked around for a while and found some similar questions but nothing that answered what I was asking for explicitly.

Answer 1

It's possible to prepare a bulk insert statement query by constructing it on the fly, but it takes a few tricks. The most important bits are using str_pad() to construct a query string of variable length, and using call_user_func_array() to call bind_param() with a variable number of parameters.

function insertBulkPrepared($db, $table, $fields, $types, $values) {
    $chunklength = 500;
    $fieldcount = count($fields);
    $fieldnames = '`'.join('`, `', $fields).'`';
    $prefix = "INSERT INTO `$table` ($fieldnames) VALUES ";
    $params = '(' . str_pad('', 3*$fieldcount - 2, '?, ') . '), ';
    $inserted = 0;

    foreach (array_chunk($values, $fieldcount*$chunklength) as $group) {
        $length = count($group);
        if ($inserted != $length) {
            if ($inserted) $stmt->close();
            $records = $length / $fieldcount;
            $query = $prefix . str_pad('', 3*$length + 2*($records - 1), $params);
            #echo "\n<br>Preparing '" . $query . "'";
            $stmt = $db->prepare($query);
            if (!$stmt) return false;
            $binding = str_pad('', $length, $types);
            $inserted = $length;
        }

        array_unshift($group, $binding);
        #echo "\n<br>Binding " . var_export($group, true);
        $bound = call_user_func_array(array($stmt, 'bind_param'), $group);
        if (!$bound) return false;
        if (!$stmt->execute()) return false;
    }

    if ($inserted) $stmt->close();
    return true;
}

This function takes your $db as a mysqli instance, a table name, an array of field names, and a flat array of references to values. It inserts up to 500 records per query, re-using prepared statements when possible. It returns true if all of the inserts succeeded, or false if any of them failed. Caveats:

• The table and field names are not escaped; I leave it up to you to ensure that they don't contain backticks. Fortunately, they should never come from user input.
• If the length of $values is not an even multiple of the length of $fields, the final chunk will probably fail at the preparation stage.
• Likewise, the length of the $types parameter should match the length of $fields in most cases, particularly when some of them differ.
• It doesn't distinguish between the three ways to fail. It also don't keep track of how many inserts succeeded, nor does it attempt to continue after an error.

With this function defined, your example code can be replaced with something like:

$inserts = array();
for ($j = 0; $j < $abilitiesMax - 2; $j++) {
    $inserts[] = &$abilityArray[$i]['match_id'];
    $inserts[] = &$abilityArray[$i]['player_slot'];
    $inserts[] = &$abilityArray[$i][$j]['ability'];
    $inserts[] = &$abilityArray[$i][$j]['time'];
    $inserts[] = &$abilityArray[$i][$j]['level'];
}

$fields = array('match_id', 'player_slot', 'ability', 'time', 'level');
$result = insertBulkPrepared($db, 'abilities', $fields, 'iiiii', $inserts);
if (!$result) {
    echo "<p>$db->error</p>";
    echo "<p>ERROR: when trying to insert abilities query</p>";
}

Those ampersands are important, because mysqli_stmt::bind_param expects references, which aren't provided by call_user_func_array in recent versions of PHP.

You didn't give us the original prepared statement, so you probably need to adjust the table and field names. It also looks like your code sits inside a loop over $i; in that case, only the for loop needs to be inside the outer loop. If you take the other lines outside the loop, you will use a bit more memory constructing the $inserts array, in return for much more efficient bulk inserts.

It's also possible to rewrite insertBulkPrepared() to accept a multi-dimensional array, eliminating one source of potential error, but that requires flattening the array after chunking it.