How to identify web-crawler?

Question

How can I filter out hits from webcrawlers etc. Hits which not is human..

I use maxmind.com to request the city from the IP.. It is not quite cheap if I have to pay

Answer 1

Checking the User-Agent will protect you from legitimate bots like Google and Yahoo.

However, if you're also being hit with spam bots, then chances are User-Agent comparison won't protect you since those bots typically forge a common User-Agent string anyway. In that instance, you would need to imploy more sophisticated measures. If user input is required, a simple image verification scheme like ReCaptcha or phpMeow will work.

If you're looking to filter out all page hits from a bot, unfortunately, there's no 100% reliable way to do this if the bot is forging its credentials. This is just an annoying fact of life on the internet that web admins have to put up with.