Detecting 'stealth' web-crawlers

Question

What options are there to detect web-crawlers that do not want to be detected?

(I know that listing detection techniques will allow the smart stealth-crawle

Answer 1

short answer: if a mid level programmer knows what he's doing you won't be able to detect a crawler without affecting the real user. Having your information publicly you won't be able to defend it against a crawler... it's like the 1st amendment right :)