问题
Will disallowing certain folder in robots.txt disallow its related subfolders? Example:
Disallow:/folder/
Will match:
/folder/page
/folder/subfolder/page
Or it will just match:
/folder/page
So if the second case is true, do I need to disallow second and subsequent subfolder separately?
Disallow: /folder/
Disallow /folder/subfolder/
Disallow /folder/subfolder/onemorefolder
回答1:
Robots.txt has no concept of "folders", it’s just strings. Whatever you specify in Disallow is the beginning of the URL path.
Disallow: / blocks any URL whose path starts with / (= all pages).
Disallow: /foo blocks any URL whose path starts with /foo:
/foo/foobar/foo.html/foo/bar/foo/bar/doe
Disallow: /foo/ blocks any URL whose path starts with /foo/:
/foo//foo/bar.html/foo/bar/foo/bar/doe
来源：https://stackoverflow.com/questions/23365010/does-related-subfolders-need-to-be-disallowed-separately-in-robots-txt