Detecting 'stealth' web-crawlers

Question

What options are there to detect web-crawlers that do not want to be detected?

(I know that listing detection techniques will allow the smart stealth-crawle

Answer 1

You can also check referrals. No referral could raise bot suspition. Bad referral means certainly it is not browser.

Adding invisible links (possibly marked as rel="nofollow"?),

* style="display: none;" on link or parent container
* placed underneath another element with higher z-index

I would'nt do that. You can end up blacklisted by google for black hat SEO :)