How to no index specific URLS?

问题

I was searching around on how to no index specific URLs but I havent found any specific info on the following.

By adding the below

<?php if(is_single(X)): ?>
<meta name="robots" content="noindex,nofollow">
<?php endif; ?>

I would be able to no index the (X) where X could be the post ID, the post title of “Hello World” for example , or a post slug of “hello-world”.

Would if be possible to specify all URLs which start with the same post slug or title for example, as in the example below?

www.test.com/REF-123-mytest.html
www.test.com/REF-123-yourtest.html
www.test.com/REF-123-histest.html

Could I leave out all URLs which start by REF-123 for example?

回答1:

By using robots.txt, you can disallow crawling.

By using meta-robots (or the HTTP header X-Robots-Tag), you can disallow indexing.

If you intend to forbid indexing, you shouldn’t disallow the URLs in robots.txt, otherwise bots will never know that you don’t want these URLs to be indexed.

In case you want to disallow crawling, you could use this robots.txt:

User-agent: *
Disallow: /REF-123

This would apply to all URLs whose paths start with REF-123 (case-sensitive!).

In case you want to disallow indexing, you could add to all these pages this meta element

<meta name="robots" content="noindex">

or send the corresponding HTTP header X-Robots-Tag:

X-Robots-Tag: noindex

回答2:

You can add this rule in a robots.txt file:

Disallow: www.test.com/REF-123*

来源：https://stackoverflow.com/questions/26860835/how-to-no-index-specific-urls