Codeigniter Allowed memory size exhausted while processing large files

问题

I'm posting this in case someone else is looking for the same solution, seeing as I just wasted two days on this bullshit.

I have a cron job that updates the database using a very large file once a day, using the following code:

if (($handle = fopen(dirname(__FILE__) . '/uncompressed', "r")) !== FALSE) 
{
    while (($data = fgets($handle)) !== FALSE) 
    {
        $thisline = json_decode($data, true);
        $this->regen($thisline);
    }
    fclose($handle);
}

This is in a Codeigniter controller that's only used for cron jobs. The $this->regen function runs through a bunch of different checks and stores the right information from the line in the database. The file itself is over 300MB of JSONs separated by newlines.

The problem: it would only process about 20,000 lines before the whole thing ran out of memory.

回答1:

I spent hours troubleshooting this and got nothing obvious. I'm using fgets, I have $query->free_result() in the right places. It didn't help. So then I started checking a loop of about 100 lines, and watched the output of memory_get_usage(). I finally narrowed it down to the Codeigniter Active Record class - every call to the class caused the memory usage to increase by a tiny amount.

Then I found this thread on Ellislabs and I got the answer. CI Active Record saves queries so that if you want to, you can build a query in multiple functions. (I am not even going to go into how dumb it is to have that switched on by default.)

Go to /config/database.php and add

$db['default']['save_queries'] = FALSE;

to the end of the file. Then make sure you build and execute queries using Active Record in a single function. If you need to switch it off just for one case, use

$this->db->save_queries = FALSE;

in the constructor or wherever you need to put it.

来源：https://stackoverflow.com/questions/22583231/codeigniter-allowed-memory-size-exhausted-while-processing-large-files