How to use XMLReader/DOMDocument with large XML file and prevent 500 error

Question

I have an XML file that is approximately 12mb which has about 16000 product's. I need to process it into a database; however, at about 6000 rows it dies with a 500 error. I'm using the Kohana framework (version 3) just in case that has anything to do with it.

Here's my code that I have inside the controller:

$xml = new XMLReader();
$xml->open("path/to/file.xml");

$doc = new DOMDocument;

// Skip ahead to the first <product>
while ($xml->read() && $xml->name !== 'product');

// Loop through <product>'s
while ($xml->name == 'product')
{
   $node = simplexml_import_dom($doc->importNode($xml->expand(), true));
   // 2 queries to database put here
   $xml->next('product');
}

The XML is a a bunch of items for a store, so the two queries are a) insert ignore the store itself and b) insert the product

Any insight would be greatly appreciated.

Answer 1

Probably you are running out of memory. Try to increase your memory limit

ini_set('memory_limit','128M');

or whatever the amount of memory is neccesary (it depends on your server). I leave you here some links with other ways of increasing the memory limit of your server:

PHP: Increase memory limit

PHP: Increase memory limit 2

Answer 2

Why are you mixing XMLReader / DomDocument? Just use XMLReader:

$reader = new XMLReader(); // initialize
$reader->open( 'file.xml' ); // open file
do {
    $sxe = simplexml_load_string( $reader->readOuterXml() ); // get current element
    echo $sxe; // echo current element
}
while( $reader->next( $this->type ) ); // repeat this for any "product" tag

The advantage of the example above is, that XMLReader will only read the current tag into memory. DomDocument reads the whole file - this is why you get error 500. With the given example you can handle XML files with hundreds of MB, without increasing your memory limit (except the current tag you try to read is greater then the available memory).