Facebook and Crawl-delay in Robots.txt?

前端 未结 5 900
旧时难觅i
旧时难觅i 2021-01-02 03:39

Does Facebook\'s webcrawling bots respect the Crawl-delay: directive in robots.txt files?

5条回答
  •  半阙折子戏
    2021-01-02 04:14

    We don't have a crawler. We have a scraper that scrapes meta data on pages that have like buttons/are shared on FB.

提交回复
热议问题