Multiple User Agents in Robots.txt

后端 未结 2 1852
慢半拍i
慢半拍i 2020-12-11 16:19

In robots.txt file I have following sections

User-Agent: Bot1
Disallow: /A

User-Agent: Bot2
Disallow: /B

User-Agent: *
Disallow: /C

Will

2条回答
  •  旧时难觅i
    2020-12-11 17:19

    If the bots obey the robots.txt file, yes the statement will be visible, so they will not be able to crawl /c.

    The wild card (*) after the user agent means all user agents.

    However bear in mind not all bots obey robots.txt

提交回复
热议问题