问题
I am looking to hide a site so that it doesn't show up in any search engine just wondering how I would go about this?
回答1:
Use a robots.txt file: http://www.google.com/support/webmasters/bin/answer.py?answer=156449
回答2:
Apart from password-protecting your site, you could add these lines to robots.txt:
User-agent: *
Disallow: /
This doesn't hide the site but rather instructs bots not to spider the content.
回答3:
You can somehow reduce your site being listed using a robots.txt. Note that this depends on the "goodwill" of the crawler, though (some spambots will explicitly look at locations that you disallow).
The only safe and reliable way of not having a site listed, sadly, is not putting it on the internet.
Simply not linking to your site will not work. Crawlers get their info from many sources, including browser referrers and domain registrars. So, in order to be "invisible", you would have to not visit your site and not register a domain (only access it via IP address).
And then, if you run your webserver based on IP address, you still have all the spambots probing random addresses. It will take a while, but they will find you.
Password protecting your site should work, effectively making it inaccessible. Though (and it is beyond my comprehension how that happens) for example there are literally thousands of ACM papers listed in Google which you cannot see without an account and logging in. Yet they are there.
回答4:
Use a robots.txt, deny from all search engines. They don't all respect robots.txt so check your server logs regularly and deny from ranges of suspected robots/crawlers:
http://httpd.apache.org/docs/2.2/howto/access.html
回答5:
You use a robots.txt file. Place the file in the root of the site with this content:
User-agent: *
Disallow: /
回答6:
most proper search engines uses bots or crawlers to websites and index them. you could Robot File method
回答7:
Have a look at nofollow Wikipedia
回答8:
You need to read about robots.txt file you are supposed to copy in your site's webroot – http://www.robotstxt.org/robotstxt.html.
来源：https://stackoverflow.com/questions/6732653/how-do-i-hide-a-site-from-search-engines