问题
The content of my site depends of cookies in the request, and when Google crawler bot visits my site it deoesn't index much content, because it does't have the specific cookies in each of its requests.
Is it possible to setup some rule that when the crawler bot is crawling my site it uses the specific cookies?
回答1:
Googlebot does not honor cookies on purpose -- it has to "see" what anybody else will see on your website, the "smallest common denominator" if you will; otherwise search results would be meaningless to an unknown amount of searchers.
Please google for "Googlebot cookies" to get pointed to discussions and documentations about search engines, how they work and why they work how they work; one solution to your problem might be to implement the "first visit/view free" rule.
回答2:
Yes, the Google crawler has the word "Googlebot" in it's request header. Simply check for that, but be warned that people can spoof this to get access to your site's content as well. As curiousguys stated in the comments, this is generally looked down upon by people who use your site and probably against Google's TOS.
来源：https://stackoverflow.com/questions/8601734/google-crawling-with-cookies