发表新帖
发表新帖

driver.page_source returns only meta name=“ROBOTS” content=“NOINDEX, NOFOLLOW” using Selenium

后端 未结
关注
1 588
予麋鹿
予麋鹿 2021-01-25 04:38

I want to scrape one website, to get the page content with this code:

from selenium import webdriver
from selenium.webdriver.common.desired_capabilities import D
相关标签:
1条回答
  • 挽巷
    2021-01-25 05:04

    <meta name="ROBOTS" content="value">

    This meta tag conveys the different search engines about the actions they are allowed and not allowed to take on a certain page. This meta tag can be placed anywhere within the <head> and </head> tags.

    Note:: As this <meta> tag does not have a site-wide effect it can contain different values on different pages of the same website.

    The valid values are:

    • Index (default value)
    • Noindex
    • None
    • Follow
    • Nofollow
    • Noarchive
    • Nosnippet

    These values can be combined as well to form the desired valid meta robots tag.

    Example:

    • <meta name="robots" content="noindex" />
    • <meta name="robots" content="index,follow" />
    • <meta name="robots" content="index,follow,noarchive" />

    content="NOINDEX, NOFOLLOW"

    The NOINDEX value conveys the search engines NOT to index the page, so the page should not show up in search results. The NOFOLLOW value conveys the search engines NOT to follow or discover the pages that are LINKED TO on this page.

    Web developers adds the NOINDEX , NOFOLLOW meta robots tag on development websites, so the search engines accidentally doesn't start sending traffic to a website that is still under construction.

    Why are you seeing?

    The reason can be either of the following:

    • You are trying to execute your auomated tests within Development Environment.
    • Development Team have accidentally added this tag to live website.
    • Development Team have forgot to remove it from live websites after going live.

    Reference

    What is the meaning of the meta name "robots" tag

    Outro

    Using the robots meta tag

    0 讨论(0)
 看不清?
提交回复
热议问题