Detecting 'stealth' web-crawlers

Question

What options are there to detect web-crawlers that do not want to be detected?

(I know that listing detection techniques will allow the smart stealth-crawle

Answer 1

One simple bot detection method I've heard of for forms is the hidden input technique. If you are trying to secure a form put a input in the form with an id that looks completely legit. Then use css in an external file to hide it. Or if you are really paranoid, setup something like jquery to hide the input box on page load. If you do this right I imagine it would be very hard for a bot to figure out. You know those bots have it in there nature to fill out everything on a page especially if you give your hidden input an id of something like id="fname", etc.