order of directives in robots.txt, do they overwrite each other or complement each other?

问题

User-agent: Googlebot
Disallow: /privatedir/

User-agent: *
Disallow: /

Now, what are disallowed for Googlebot: /privatedir/, or the whole website / ?

回答1:

According to the original robots.txt specification:

1. A bot must follow the first record that matches its user-agent name.

2. If such a record doesn’t exist, it must follow the record with User-agent: * (this line may not appear in more than one record).

3. If such a record doesn’t exist, it doesn’t have to follow any record.

So a bot never follows more than one record.

---

For your example this means:

• A bot that matches the name "Googlebot" is not allowed to crawl URLs with a path that starts with /privatedir/.
• A bot that doesn’t match the name "Googlebot" is not allowed to crawl any URL.

来源：https://stackoverflow.com/questions/45293419/order-of-directives-in-robots-txt-do-they-overwrite-each-other-or-complement-ea