问题
Where should put robots.txt?
domainname.com/robots.txt
or
domainname/public_html/robots.txt
I placed the file in domainname.com/robots.txt, but it's not opening when I type this in browser.
alt text http://shup.com/Shup/358900/11056202047-My-Desktop.png
回答1:
Where the file goes in your filesystem depends on what host you're using, so it's hard for us to give a specific answer about that.
The best description is: put it wherever the index.html (or index.php or whatever) file is that represents your homepage. If that's domainname/public_html/index.html, for example, put it in domainname/public_html/robots.txt.
回答2:
i think the better way to describe it is to have it in the root web folder of your domain... so http://example.com/robots.txt you can also put your sitemap.xml in the root or refer to it with a Sitemap: http://example.com/fldr/smap.xml line in your robots.txt.
dont forget: you can use Google Webmaster Tools to check to make sure you haven't restricted anything you didnt mean to(you also get to see queries and links woohoo!).
suggestion: id consider using the <META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW"> if possible because you will still earn linkjuice for links on the page but it wont show up in googles index while a robots.txt directive can leave a plain url with do description in SERPs but will loose all value of links pointed to it because its robots.txted out (its ranking b/c of anchor text so get credit for it)
回答3:
In the root of your web directory (where you put the files that show up on your website)
回答4:
In this case you should put it in domainname/public_html/robots.txt, as the public.html folder is where your index file will be.
来源：https://stackoverflow.com/questions/2984607/where-to-put-robots-txt-file