Where should put robots.txt?
domainname.com/robots.txt
or
domainname/public_html/robots.txt
I placed the file in domainname.com/robots.txt, but it's not opening when I type this in browser.
alt text http://shup.com/Shup/358900/11056202047-My-Desktop.png
Where the file goes in your filesystem depends on what host you're using, so it's hard for us to give a specific answer about that.
The best description is: put it wherever the index.html (or index.php or whatever) file is that represents your homepage. If that's domainname/public_html/index.html, for example, put it in domainname/public_html/robots.txt.
i think the better way to describe it is to have it in the root web folder of your domain... so http://example.com/robots.txt you can also put your sitemap.xml in the root or refer to it with a Sitemap: http://example.com/fldr/smap.xml line in your robots.txt.
dont forget: you can use Google Webmaster Tools to check to make sure you haven't restricted anything you didnt mean to(you also get to see queries and links woohoo!).
suggestion: id consider using the <META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW"> if possible because you will still earn linkjuice for links on the page but it wont show up in googles index while a robots.txt directive can leave a plain url with do description in SERPs but will loose all value of links pointed to it because its robots.txted out (its ranking b/c of anchor text so get credit for it)
In the root of your web directory (where you put the files that show up on your website)
In this case you should put it in domainname/public_html/robots.txt, as the public.html folder is where your index file will be.
来源：https://stackoverflow.com/questions/2984607/where-to-put-robots-txt-file