问题
I have a github page from my repository username.github.io
However I do not want Google to crawl my website and absolutely do not want it to show up on search results.
Will just using robots.txt in github pages work? I know there are tutorials for stop indexing Github repository but what about the actual Github page?
回答1:
Will just using robots.txt in github pages work?
If you're using the default GitHub Pages subdomain, then no because Google would check https://github.io/robots.txt only.
You can make sure you don't have a master branch, or that your GitHub repo is a private one, although, as commented by olavimmanuel and detailed in olavimmanuel's answer, this would not change anything.
However, if you're using a custom domain with your GitHub Pages site, you can place a robots.txt file at the root of your repo and it will work as expected. One example of using this pattern is the repo for Bootstrap.
回答2:
I don't know if it is still relevant, but google says you can stop spiders with a meta tag:
<meta name="robots" content="noindex">
I'm not sure however if that works for all spiders or google only.
回答3:
Short answer:
You can use a robots.txt to stop indexing of your users GitHub Pages by adding it in your User Page. This robots.txt will be the active robots.txt for all your projects pages as the project pages are reachable as subdirectories (username.github.io/project) in your subdomain (username.github.io).
Longer answer:
You get your own subdomain for GitHub pages (username.github.io). According to this question on MOZ and googles reference each subdomain has/needs its own robots.txt.
This means that the valid/active robots.txt for project projectname by user username lives at username.github.io/robots.txt. You can put a robots.txtfile there by creating a GitHub Pages page for your user.
This is done by creating a new project/repository named username.github.io where username is your username. You can now create a robots.txt file in the master branch of this project/repository and it should be visible at username.github.io/robots.txt. More information about project, user and organization pages can be found here.
I have tested this with Google, confirming ownership of myusername.github.io by placing a html file in my project/repository https://github.com/myusername/myusername.github.io/tree/master, creating a robot.txt file there and then verifying that my robots.txt works by using Googles Search Console webmaster tools (googlebot-fetch). Google does indeed list it as blocked and Google Search Console webmaster tools (robots-testing-tool) confirms it.
To block robots for one projects GitHub Page:
User-agent: *
Disallow: /projectname/
To block robots for all GitHub Pages for your user (User Page and all Project Pages):
User-agent: *
Disallow: /
Other options
- Look into the HTML meta tag
- Look into custom domain (redirects) for GitHub Pages
回答4:
Google doesn't recommend using a robots.txt file to not index a website(GitHub page in this case). In fact most of the time it does get indexed even if you block the google bot.
Instead, you should add the following in your page head, which should be easy to control even if you are not using a custom domain.
<meta name='robots' content='noindex,nofollow' />
This will tell Google to NOT index it. Where if you only block google bot to access your website it will still index like 90% of the time just won't show meta description.
来源：https://stackoverflow.com/questions/32784322/stopping-index-of-github-pages