Importing Large XML file into SQL 2.5Gb

Question

Hi I am trying to import a large XML file into a table on my sql server (2014)

I have used the code below for smaller files and thought it would be ok as this is a o

Answer 1

Try this. Just another method that I have used for some time. It's pretty fast (could be faster). I pull a huge xml db from a gaming company every night. This is how i get it an import it.

 $xml  = new XMLReader();            
 $xml->open($xml_file); // file is your xml file you want to parse
 while($xml->read() && $xml->name != 'game') { ; } // get past the header to your first record (game in my case)

while($xml->name == 'game') { // now while we are in this record               
                $element        = new SimpleXMLElement($xml->readOuterXML());
                $gameRec        = $this->createGameRecord($element, $os); // this is my function to reduce some clutter - and I use it elsewhere too

                /* this looks confusing, but it is not. There are over 20 fields, and instead of typing them all out, I just made a string. */
                $sql = "INSERT INTO $table (";
                foreach($gameRec as $field=>$game){
                $sql .= " $field,";
                }
                $sql = rtrim($sql, ",");
                $sql .=") values (";

                foreach($gameRec as $field=>$game) {
                    $sql .= " :$field,";               
                }
                $sql = rtrim($sql,",");
                $sql .= ") ON DUPLICATE KEY UPDATE "; // online game doesn't have a gamerank - not my choice LOL, so I adjust that for here

                switch ($os) {
                    case 'pc' : $sql .= "gamerank = ".$gameRec['gamerank']        ; break;
                    case 'mac': $sql .= "gamerank = ".$gameRec['gamerank']        ; break;
                    case 'pl' : $sql .= "playercount = ".$gameRec['playercount']  ; break;
                    case 'og' :
                        $playercount = $this->getPlayerCount($gameRec['gameid']);
                        $sql .= "playercount = ".$playercount['playercount']  ;
                        break;

                }


                try {

                    $stmt = $this->connect()->prepare($sql);
                    $stmt->execute($gameRec);

                } catch (PDOException $e) {// Kludge

                    echo 'os: '.$os.'
table: '.$table.'
XML LINK: '.$comprehensive_xml.'
Current Record:
'.print_r($gameRec).'

'.
                    'SQL: '.$sql.'
';
                    die('Line:33
Function: pullBFG()
Cannot add game record 
'.$e->getMessage());

                }

                /// VERY VERY VERY IMPORTANT do not forget these 2 lines, or it will go into a endless loop - I know, I've done it. locks up your system after a bit hahaah
                $xml->next('game');
                unset($element);
            }// while there are games

This should get you started. Obviously, adjust the "game" to your xml records. Trim out the fat I have here.

Here is the createGameRecord($element, $type='pc') Basically it turns it into an array to use elsewhere, and makes it easier to add it to the db. with a single line as seen above: $stmt->execute($gameRec); Where $gameRec was returned from this function. PDO knows gameRec is an array, and will parse it out as you INSERT IT. the "delHardReturns() is another of my fucntion that gets rid of those hard returns /r /n etc.. Seems to mess up the SQL. I think SQL has a function for that, but I have not pursed it. Hope you find this useful.

private function createGameRecord($element, $type='pc') {
            if( ($type == 'pc') || ($type == 'og') ) { // player count is handled separately
                $game = array(
                    'gamename'                  => strval($element->gamename),
                    'gameid'                    => strval($element->gameid),                
                    'genreid'                   => strval($element->genreid),
                    'allgenreid'                => strval($element->allgenreid),
                    'shortdesc'                 => $this->delHardReturns(strval($element->shortdesc)),
                    'meddesc'                   => $this->delHardReturns(strval($element->meddesc)),
                    'bullet1'                   => $this->delHardReturns(strval($element->bullet1)),
                    'bullet2'                   => $this->delHardReturns(strval($element->bullet2)),
                    'bullet3'                   => $this->delHardReturns(strval($element->bullet3)),
                    'bullet4'                   => $this->delHardReturns(strval($element->bullet4)),
                    'bullet5'                   => $this->delHardReturns(strval($element->bullet5)),
                    'longdesc'                  => $this->delHardReturns(strval($element->longdesc)),
                    'foldername'                => strval($element->foldername),
                    'hasdownload'               => strval($element->hasdownload),
                    'hasdwfeature'              => strval($element->hasdwfeature),                             
                    'releasedate'               => strval($element->releasedate)

                );

                if($type === 'pc')  {

                    $game['hasvideo']           = strval($element->hasvideo);
                    $game['hasflash']           = strval($element->hasflash);
                    $game['price']              = strval($element->price); 
                    $game['gamerank']           = strval($element->gamerank);
                    $game['gamesize']           = strval($element->gamesize);
                    $game['macgameid']          = strval($element->macgameid);
                    $game['family']             = strval($element->family);
                    $game['familyid']           = strval($element->familyid);
                    $game['productid']          = strval($element->productid);
                    $game['pc_sysreqos']        = strval($element->systemreq->pc->sysreqos);
                    $game['pc_sysreqmhz']       = strval($element->systemreq->pc->sysreqmhz);
                    $game['pc_sysreqmem']       = strval($element->systemreq->pc->sysreqmem);
                    $game['pc_sysreqhd']        = strval($element->systemreq->pc->sysreqhd);

                    if(empty($game['gamerank'])) $game['gamerank'] = 99999;

                    $game['gamesize'] = $this->readableBytes((int)$game['gamesize']);  


                }// dealing with PC type

                if($type === 'og') {
                    $game['onlineiframeheight']              = strval($element->onlineiframeheight);
                    $game['onlineiframewidth']              = strval($element->onlineiframewidth); 

                }

                $game['releasedate']            = substr($game['releasedate'],0,10);

            } else {// not type = pl

                $game['playercount']            = strval($element->playercount);
                $game['gameid']                 = strval($element->gameid);
            }// no type = pl else


            return $game;
        }/