Is the User-Agent line in robots.txt an exact match or a substring match?

后端 未结 2 1613
你的背包
你的背包 2020-12-21 04:30

When a crawler reads the User-Agent line of a robots.txt file, does it attempt to match it exactly to its own User-Agent or does it attempt to match it as a substring of its

2条回答
  •  被撕碎了的回忆
    2020-12-21 04:56

    In the original robots.txt specification (from 1994), it says:

    User-agent

    […]

    The robot should be liberal in interpreting this field. A case insensitive substring match of the name without version information is recommended.

    […]

    If and which bots/parsers comply with this is another question and can’t be answered in general.

提交回复
热议问题