How to prevent search engines from indexing a single page of my website?

自闭症网瘾萝莉.ら 提交于 2019-11-28 06:19:35

You need a simple robots.txt file. Basically, it's a text file that tells search engines not to index particular pages.
You don't need to include it in the header of your page; as long as it's in the root directory of your website it will be picked up by crawlers.
Create it in the root folder of your website and put the following text in:

User-Agent: *
Disallow: /imprint-page.htm

Note that you'd replace imprint-page.html in the example with the actual name of the page (or the directory) that you wish to keep from being indexed.

That's it! If you want to get more advanced, you can check out here, here, or here for a lot more info. Also, you can find free tools online that will generate a robots.txt file for you (for example, here).

Also you can add following meta tag in HEAD of that page

<meta name="robots" content="noindex,nofollow" />

You can setup a robots.txt file to try and tell search engines to ignore certain directories.

See here for more info.

Basically:

User-agent: *
Disallow: /[directory or file here]
Tahir Afridi
<meta name="robots" content="noindex, nofollow">

Just include this line in your <html> <head> tag. Why I'm telling you this because if you use robots.txt file to hide your URLs that might be login pages or other protected URLs that you won't show to someone else or search engines.

What I can do is just accessing the robots.txt file directly from your website and can see which URLs you have are secret. Then what is the logic behind this robots.txt file?

The good way is to include the meta tag from above and keep yourself safe from anyone.

Jérôme Verstrynge

Nowadays, the best method is to use a robots meta tag and set it to noindex,follow:

<meta name="robots" content="noindex, follow">

Create a robots.txt file and set the controls there.

Here are the docs for google: http://code.google.com/web/controlcrawlindex/docs/robots_txt.html

A robot wants to vists a Web site URL, say http://www.example.com/welcome.html. Before it does so, it firsts checks for http://www.example.com/robots.txt, and finds: you can explicitly disallow :

User-agent: *
Disallow: /~joe/junk.html

please visit below link for details robots.txt

易学教程内所有资源均来自网络或用户发布的内容,如有违反法律规定的内容欢迎反馈
该文章没有解决你所遇到的问题?点击提问,说说你的问题,让更多的人一起探讨吧!