问题
This bot doesn't respect nofollow noindex in robots.txt.
I have this in robots.txt:
User-agent: Msnbot
Disallow: /
User-Agent: Msnbot/2.0b
Disallow: /
Till now it was pretty slow, but now, it is a monster that won't leave my site at all. Crawls all WordPress and MyBB 24/7.
To block IP ranges or what can I do to stop all of this content stealers?
回答1:
Based on Block by useragent or empty referer you could something like this in your .htaccess
Options +FollowSymlinks
RewriteEngine On
RewriteBase /
SetEnvIfNoCase User-Agent "^Msnbot" ban_agent
Deny from env=ban_agent
回答2:
Here's what you need to do instead:
Code:
User-agent: *
Disallow:
User-agent: MSNbot
Disallow: /
The above code allows all robots except MSNbot.
You can read more about the robots exclusion protocol here.
for example, for bing.
User-agent: MSNBot
Disallow: /
for google
User-agent: googlebot
Disallow: /
if you want block all bots. use this.
User-agent: *
Disallow: /
来源：https://stackoverflow.com/questions/22128434/how-to-block-a-bot-that-is-excessively-visiting-my-site