问题
I am using this code in my robots.txt file:
User-agent: *
Disallow:
But, my one competitor is using
User-agent: *
Disallow: /
And his site is performing well in Google and on 1st rank. But my site is not ranking. I have checked everything in my site and it is OK. But how my competitor is performing well in Google? Is robots.txt file doesn't matter?
回答1:
Your site ranking is not depended on your robots.txt file. You just tell google how to crawl and index pages on your website.
Web site owners use the /robots.txt file to give instructions about their site to web robots;
Your site ranking depends on many factors like:
- Keyword usage
- Site structure
- Site speed
- Time spent on site
- Number of inbound links
- Quality of inbound links
By the way this will block all web crawlers from all content:
User-agent: *
Disallow: /
And this will allow all crawlers to access everything:
User-agent: *
Allow: /
回答2:
Please follow as below steps.
1)
User-agent: *
Disallow: /
Add which page or direct you don't want to crawl if you're targeting main director add public/domain name in it for disallow all
2)our site ranking depends on many factors like: <- This thing also matters nice wrote.
Keyword usage
Site structure
Site speed
Time spent on site
Number of inbound links
Quality of inbound links
3)take a help for website source http://www.robotsgenerator.com/
P.S. :- please remove website owner marketing link in robots.txt file for sure.
来源：https://stackoverflow.com/questions/30681245/robots-txt-file