Robots.txt Disallow Certain Folder Names

问题

I want to disallow robots from crawling any folder, at any position in the url with the name: this-folder.

Examples to disallow:

http://mysite.com/this-folder/
http://mysite.com/houses/this-folder/
http://mysite.com/some-other/this-folder/
http://mysite.com/no-robots/this-folder/

This is my attempt:

Disallow: /.*this-folder/

Will this work?

回答1:

Officially globbing and regular expressions are not supported:

http://www.robotstxt.org/robotstxt.html

but apparently some search engines support this.

来源：https://stackoverflow.com/questions/3501661/robots-txt-disallow-certain-folder-names