I have a website I am developing that is also going to be pulled into a web app. I have the following code in my .htaccess file to prevent access from ANYONE th
.htaccess
SetEnvIfNoCase User-Agent .*google.* search_robot SetEnvIfNoCase User-Agent .*yahoo.* search_robot SetEnvIfNoCase User-Agent .*bot.* search_robot SetEnvIfNoCase User-Agent .*ask.* search_robot Order Deny,Allow Deny from All Allow from env=search_robot
Htaccess SetEnvIf and SetEnvIfNoCase Examples