robots.txt

My robots.txt in Google Webmaster Tools shows the following values: User-agent: * Allow: / What does it mean? I don't have enough knowledge about it, so looking for your help. I want to allow all robots to crawl my website, is this the right configuration? That file will allow all crawlers access User-agent: * Allow: / This basically allows all user agents (the *) to all parts of the site (the /). If you want to allow every bot to crawl everything, this is the best way to specify it in your robots.txt: User-agent: * Disallow: Note that the Disallow field has an empty value, which means

I was unable to find information about my case. I want to restrict the following types of URLs to be indexed: website.com/video-title/video-title/ (my website produces such double URL copies of my video-articles) Each video article starts with the word "video" in the beginning of its URL. So what I want to do is to restrict all URLs that have website.com/"any-url"/video-any-url" This way I will remove all the doubled copies. Could somebody help me? This is not possible in the original robots.txt specification. But some parsers may support wildcards in Disallow anyway, for example, Google :

Will this robots.txt file only allow googlebot to index my site's index.php file? CAVEAT, I have an htaccess redirect that people who type in http://www.example.com/index.php are redirected to simply http://www.example.com/ So, this is my robots.txt file content... User-agent: Googlebot Allow: /index.php Disallow: / User-agent: * Disallow: / Thanks in advance! Not really. Good bots Only "good" bots follow the robots.txt instructions (not all robots and spiders bother to read/follow robots.txt ). That might not even include all the main search engine's bots, but it definitely mean that some web

I just ran into a robots.txt that looks like this: User-agent: * Disallow: /foobar User-agent: badbot Disallow: * After disallowing only a few folders for all, does the specific badbot rule even apply? Note: This question is merely for understanding the above ruleset. I know using robots.txt is not a proper security mechanism and I'm neither using nor advocating it. unor Each bot only ever complies to at most a single record (block). A block starts with one or more User-agent lines, typically followed by Disallow lines ( at least one is required ). Blocks are separated by blank lines. A bot

I've published a website and, due to a misunderstanding not depending on me, I had to block all the pages before indexing. Some of these pages had been already linked on social networks, so to avoid a bad user-experience I've decided to insert the following code into "robots.txt" User-agent: * Disallow: * I've received a "critical problem" alert on webmaster tools and I'm a bit worried about it. In your experience, would it be sufficient (whenever possible) to restore the original "robots.txt"? May the current situation leave consequences (penalizations or similar) on the website if it lasts

Is this a good idea?? http://browsers.garykeith.com/stream.asp?RobotsTXT What does abusive crawling mean? How is that bad for my site? Not really. Most "bad bots" ignore the robots.txt file anyway. Abuse crawling usually means scraping. These bots are showing up to harvest email addresses or more commonly, content. As to how you can stop them? That's really tricky and often not wise. Anti-crawl techniques have a tendency to be less than perfect and cause problems for regular humans. Sadly, like "shrinkage" in retail, it's a cost of doing business on the web. A user-agent (which includes