Ruby on Rails, How to determine if a request was made by a robot or search engine spider?

Question

I\'ve Rails apps, that record an IP-address from every request to specific URL, but in my IP database i\'ve found facebook blok IP like 66.220.15.* and Google IP (i suggest

Answer 1

Since the well behaved bots at least typically include a reference URI in the UA string they send, something like:

request.env["HTTP_USER_AGENT"].match(/\(.*https?:\/\/.*\)/)

is an easy way to see if the request is from a bot vs. a human user's agent. This seems to be more robust than trying to match against a comprehensive list.