Are Robots.txt and metadata tags enough to stop search engines to index dynamic pages that are dependent of $_GET variables?

前端 未结 2 1994
失恋的感觉
失恋的感觉 2020-12-12 02:35

I created a php page that is only accessible by means of token/pass received through $_GET

Therefore if you go to the following url you\'ll get a generic or blank pag

2条回答
  •  心在旅途
    2020-12-12 02:56

    If a search engine bot finds the link with the token somehow¹, it may crawl and index it.

    If you use robots.txt to disallow crawling the page, conforming search engine bots won’t crawl the page, but they may still index its URL (which then might appear in a site: search).

    If you use meta-robots to disallow indexing the page, conforming search engine bots won’t index the page, but they may still crawl it.

    You can’t have both: If you disallow crawling, conforming bots can never learn that you also disallow indexing, because they are not allowed to visit the page to see your meta-robots element.

    ¹ There are countless ways how search engines might find a link. For example, a user that visits the page might use a browser toolbar that automatically sends all visited URLs to a search engine.

提交回复
热议问题