Robots.txt Allow sub folder but not the parent

前端 未结 3 809
南旧
南旧 2021-01-03 20:54

Can anybody please explain the correct robots.txt command for the following scenario.

I would like to allow access to:

/directory/subdirec

3条回答
  •  爱一瞬间的悲伤
    2021-01-03 21:44

    Be aware that there is no real official standard and that any web crawler may happily ignore your robots.txt

    According to a Google groups post, the following works at least with GoogleBot;

    User-agent: Googlebot 
    Disallow: /directory/ 
    Allow: /directory/subdirectory/
    

提交回复
热议问题