Best practice: Import mySQL file in PHP; split queries

Question

I have a situation where I have to update a web site on a shared hosting provider. The site has a CMS. Uploading the CMS\'s files is pretty straightforward using FTP.

<

Answer 1

Can't you install phpMyAdmin, gzip the file (which should make it much smaller) and import it using phpMyAdmin?

EDIT: Well, if you can't use phpMyAdmin, you can use the code from phpMyAdmin. I'm not sure about this particular part, but it's generaly nicely structured.

Answer 2

Already answered: Loading .sql files from within PHP Also:

• http://webxadmin.free.fr/article/import-huge-mysql-dumps-using-php-only-342.php
• http://www.phpbuilder.com/board/showthread.php?t=10323180
• http://forums.tizag.com/archive/index.php?t-3581.html

Answer 3

When StackOverflow released their monthly data dump in XML format, I wrote PHP scripts to load it into a MySQL database. I imported about 2.2 gigabytes of XML in a few minutes.

My technique is to prepare() an INSERT statement with parameter placeholders for the column values. Then use XMLReader to loop over the XML elements and execute() my prepared query, plugging in values for the parameters. I chose XMLReader because it's a streaming XML reader; it reads the XML input incrementally instead of requiring to load the whole file into memory.

You could also read a CSV file one line at a time with fgetcsv().

If you're inporting into InnoDB tables, I recommend starting and committing transactions explicitly, to reduce the overhead of autocommit. I commit every 1000 rows, but this is arbitrary.

I'm not going to post the code here (because of StackOverflow's licensing policy), but in pseudocode:

connect to database
open data file
PREPARE parameterizes INSERT statement
begin first transaction
loop, reading lines from data file: {
    parse line into individual fields
    EXECUTE prepared query, passing data fields as parameters
    if ++counter % 1000 == 0,
        commit transaction and begin new transaction
}
commit final transaction

Writing this code in PHP is not rocket science, and it runs pretty quickly when one uses prepared statements and explicit transactions. Those features are not available in the outdated mysql PHP extension, but you can use them if you use mysqli or PDO_MySQL.

I also added convenient stuff like error checking, progress reporting, and support for default values when the data file doesn't include one of the fields.

I wrote my code in an abstract PHP class that I subclass for each table I need to load. Each subclass declares the columns it wants to load, and maps them to fields in the XML data file by name (or by position if the data file is CSV).

Answer 4

You can use phpMyAdmin for importing the file. Even if it is huge, just use UploadDir configuration directory, upload it there and choose it from phpMyAdmin import page. Once file processing will be close to the PHP limits, phpMyAdmin interrupts importing, shows you again import page with predefined values indicating where to continue in the import.

Answer 5

Single page PHPMyAdmin - Adminer - Just one PHP script file. check : http://www.adminer.org/en/

Answer 6

First at all thanks for this topic. This saved a lot of time for me :) And let me to make little fix for your code. Sometimes if TRIGGERS or PROCEDURES is in dump file, it is not enough to examine the ; delimiters. In this case may be DELIMITER [something] in sql code, to say that the statement will not end with ; but [something]. For example a section in xxx.sql:

    DELIMITER //
    CREATE TRIGGER `mytrigger` BEFORE INSERT ON `mytable`
    FOR EACH ROW BEGIN
         SET NEW.`create_time` = NOW();
    END
    //
    DELIMITER ;

So first need to have a falg, to detect, that query does not ends with ; And delete the unqanted query chunks, because the mysql_query does not need delimiter (the delimiter is the end of string) so mysql_query need someting like this:

    CREATE TRIGGER `mytrigger` BEFORE INSERT ON `mytable`
    FOR EACH ROW BEGIN
         SET NEW.`create_time` = NOW();
    END;

So a little work and here is the fixed code:

    function SplitSQL($file, $delimiter = ';')
    {
        set_time_limit(0);            
        $matches = array();
        $otherDelimiter = false;
        if (is_file($file) === true) {
            $file = fopen($file, 'r');
            if (is_resource($file) === true) {
                $query = array();
                while (feof($file) === false) {
                    $query[] = fgets($file);
                    if (preg_match('~' . preg_quote('delimiter', '~') . '\s*([^\s]+)$~iS', end($query), $matches) === 1){     
                        //DELIMITER DIRECTIVE DETECTED
                        array_pop($query); //WE DON'T NEED THIS LINE IN SQL QUERY
                        if( $otherDelimiter = ( $matches[1] != $delimiter )){
                        }else{
                            //THIS IS THE DEFAULT DELIMITER, DELETE THE LINE BEFORE THE LAST (THAT SHOULD BE THE NOT DEFAULT DELIMITER) AND WE SHOULD CLOSE THE STATEMENT                                
                            array_pop($query);
                            $query[]=$delimiter;
                        }                                                                                    
                    }                        
                    if ( !$otherDelimiter && preg_match('~' . preg_quote($delimiter, '~') . '\s*$~iS', end($query)) === 1) {                            
                        $query = trim(implode('', $query));
                        if (mysql_query($query) === false){
                            echo '<h3>ERROR: ' . $query . '</h3>' . "\n";
                        }else{
                            echo '<h3>SUCCESS: ' . $query . '</h3>' . "\n";
                        }
                        while (ob_get_level() > 0){
                            ob_end_flush();
                        }
                        flush();                        
                    }
                    if (is_string($query) === true) {
                        $query = array();
                    }
                }                    
                return fclose($file);
            }
        }
        return false;
}

I hope i could help somebody too. Have a nice day!