Does related subfolders need to be disallowed separately in robots.txt?

Question

Will disallowing certain folder in robots.txt disallow its related subfolders? Example:

Disallow:/folder/

Will match:

/folder/p         


        

Answer 1

Robots.txt has no concept of "folders", it’s just strings. Whatever you specify in Disallow is the beginning of the URL path.

Disallow: / blocks any URL whose path starts with / (= all pages).

Disallow: /foo blocks any URL whose path starts with /foo:

• /foo
• /foobar
• /foo.html
• /foo/bar
• /foo/bar/doe

Disallow: /foo/ blocks any URL whose path starts with /foo/:

• /foo/
• /foo/bar.html
• /foo/bar
• /foo/bar/doe