robots.txt

I am trying to block a couple bots via my htaccess file. On Search Engine Watch it is recommended to use the below. I did block these bots in the robots.txt file but they are ignoring it. Here is code from Search Engine Watch: RewriteEngine on Options +FollowSymlinks RewriteBase / RewriteCond %{HTTP_USER_AGENT} ^Baiduspider [NC,OR” RewriteCond %{HTTP_USER_AGENT} ^Sogou RewriteRule ^.*$ - [F” My current htaccess file is as below. How exactly would I modify my current .htaccess with the above code? What do I add and where do I add the above? I assume some of the above is already included in my

I have urls like these: /products/:product_id/deals/new /products/:product_id/deals/index I'd like to disallow the "deals" folder in my robots.txt file. [Edit] I'd like to disallow this folder for Google, Yahoo and Bing Bots. Does anyone know if these bots support wildcard character and so would support the following rule? Disallow: /products/*/deals Also... Do you have any really good tuto on robots.txt rules? As I didn't manage to find a "really" good one I could use one... And one last question: Is the robots.txt the best way to handle this? Or should I better use the "noindex" meta? Thx

I have a small situation where i have to remove my robots.txt file because i don't want and robots crawlers to get the links. Also i want them to be accessible by the user and i don't want them to be cached by the search engines. Also i cannot add any user authentications for various reasons. So i am thinking about using mod-rewrite to disable search engine crawlers from crawling it while allowing all others to do it. The logic i am trying to implement is write a condition to check if the incomming user agent is a search engine and if yes then re-direct them to 401. The only problem is i don't

As traffic is distributed unevenly over 24 hours, I would like to disallow crawlers during peek hours and allow them at non-busy hours. Is there a method to achieve this? edit: thanks for all the good advice. This is another solution we found. 2bits.com has an article on setting IPTables firewall to limit the number of connections from certain IP addresses. the article the setting of IPTables: Using connlimit In newer Linux kernels, there is a connlimit module for iptables. It can be used like this: iptables -I INPUT -p tcp -m connlimit --connlimit-above 5 -j REJECT This limits connections