Stop abusive bots from crawling?

拜拜、爱过 提交于 2019-12-01 21:49:06

Not really. Most "bad bots" ignore the robots.txt file anyway.

Abuse crawling usually means scraping. These bots are showing up to harvest email addresses or more commonly, content.

As to how you can stop them? That's really tricky and often not wise. Anti-crawl techniques have a tendency to be less than perfect and cause problems for regular humans.

Sadly, like "shrinkage" in retail, it's a cost of doing business on the web.

A user-agent (which includes crawlers) is under no obligation to honour your robots.txt. The best you can do is try to identify abusive access patterns (via web-logs, etc.), and block the corresponding IP.

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!