How to add `nofollow, noindex` all pages in robots.txt?

Question

I want to add nofollow and noindex to my site whilst it's being built. The client has request I use these rules.

I am aware of

<meta name="robots" content="noindex,nofollow">

But I only have access to the robots.txt file.

Does anyone know the correct format I can use to apply noindex, nofollow rules via the robots.txt file?

Answer 1

noindex and nofollow means you do not want your site to crawl in search engine.

so simply put code in robots.txt User-agent: * Disallow: /

it means noindex and nofollow.

Answer 2

There is a non-standard Noindex field, which Google (and likely no other consumer) supported as experimental feature.

Following the robots.txt specification, you can’t disallow indexing nor following links with robots.txt.

For a site that is still in development, has not been indexed yet, and doesn’t get backlinks from pages which may be crawled, using robots.txt should be sufficient:

# no bot may crawl 
User-agent: *
Disallow: /

If pages from the site are already indexed, and/or if other pages which may be crawled link to it, you have to use noindex, which can not only be specified in the HTML, but also as HTTP header:

X-Robots-Tag: noindex, nofollow

Answer 3

• Noindex tells search engines not to include pages in search results, but can follow links (and also can transfer PA and DA)
• Nofollow tells bots not to follow the links. We also can combine noindex with follow in pages we don´t want to be indexed, but we want to follow the links