Why does readfile() exhaust PHP memory?

Question

I've seen many questions about how to efficiently use PHP to download files rather than allowing direct HTTP requests (to keep files secure, to track downloads, etc.).

The answer is almost always PHP readfile().

• Downloading large files reliably in PHP
• How to force download of big files without using too much memory?
• Best way to transparently log downloads?

BUT, although it works great during testing with huge files, when it's on a live site with hundreds of users, downloads start to hang and PHP memory limits are exhausted.

So what is it about how readfile() works that causes memory to blow up so bad when traffic is high? I thought it's supposed to bypass heavy use of PHP memory by writing directly to the output buffer?

EDIT: (To clarify, I'm looking for a "why", not "what can I do". I think that Apache's mod_xsendfile is the best way to circumvent)

Answer 1

Description
int readfile ( string $filename [, bool $use_include_path = false [, resource $context ]] )
Reads a file and writes it to the output buffer*.

PHP has to read the file and it writes to the output buffer. So, for 300Mb file, no matter what the implementation you wrote (by many small segments, or by 1 big chunk) PHP has to read through 300Mb of file eventually.

If multiple user has to download the file, there will be a problem. (In one server, hosting providers will limit memory given to each hosting user. With such limited memory, using buffer is not going to be a good idea. )

I think using the direct link to download a file is a much better approach for big files.

Answer 2

If you have output buffering on than use ob_end_flush() right before the call to readfile()

header(...);
ob_end_flush();
@readfile($file);

Answer 3

You might want to turn off output buffering altogether for that particular location, using PHP's output_buffering configuration directive.

Apache example:

<Directory "/your/downloadable/files">
     ...
     php_admin_value output_buffering "0"
     ...
</Directory>

"Off" as the value seems to work as well, while it really should throw an error. At least according to how other types are converted to booleans in PHP. *shrugs*

Answer 4

Came up with this idea in the past (as part of my library) to avoid high memory usage:

function suTunnelStream( $sUrl, $sMimeType, $sCharType = null )
{
  $f = @fopen( $sUrl, 'rb' );
  if( $f === false )
   { return false; }

  $b = false;
  $u = true;

  while( $u !== false && !feof($f ))
  { 
    $u = @fread( $f, 1024 );
    if( $u !== false )
    {  
      if( !$b )
       { $b = true;
         suClearOutputBuffers();
         suCachedHeader( 0, $sMimeType, $sCharType, null, !suIsValidString($sCharType)?('content-disposition: attachment; filename="'.suUniqueId($sUrl).'"'):null );
       } 
      echo $u; 
    }
 } 

  @fclose( $f );
  return ( $b && $u !== false );
}

Maybe this can give you some inspiration.

Answer 5

As mentioned here: "Allowed memory .. exhausted" when using readfile, the following block of code at the top of the php file did the trick for me.

This will checks if php output buffering is active. If so it turns it off.

if (ob_get_level()) {
    ob_end_clean();
}

Answer 6

Well, it is memory intensive function. I would pipe users to a static server that has specific rule set in place to control downloads instead of using readfile().

If that's not an option add more RAM to satisfy the load or introduce queuing system that gracefully controls server usage.