I have a php file that acts as a gatekeeper for all the files I want people to download, who ahve sufficient privilages.
The code I use throw the file to the user i
You'll want to somehow authenticate them (an HTML form, HTTP basic auth, whatever), then set a session flag, which your download.php script can check. Note that this doesn't prevent people from downloading the file, then distributing it themselves.
You should configure your web server so the real files are not directly accessible.
It's not going to cause memory problems per se. readfile does not read the file into memory. However, using PHP will create overhead. You can eliminate some of this delay by using X-Sendfile.