sogou spider still hitting our website even after blocking it

问题

Our website was getting many hits from "Sogou web spider", So we thought of blocking it using htaccess rules. We created below rules -

RewriteCond %{HTTP_USER_AGENT} Sogou [NC]
RewriteRule ^.*$ - [L]

However we are still getting hits from Sogou. I would like to know what changes should I make in this rule to block Sogou.

Thanking you,

回答1:

As @faa mentioned, you're not actually blocking anything:

RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Sogou [NC]
RewriteRule ^.*$ map.txt [R=403]

Make sure you've got RewriteEngine On and the [R=403].

You may still see hits from them in your access logs but with the combination of not sending any data and a 403 forbidden header, you should see the hits die off eventually. Even if they continue to crawl your site, it should no longer generate so much extra load on your server.

来源：https://stackoverflow.com/questions/14580854/sogou-spider-still-hitting-our-website-even-after-blocking-it