robots.txt

I have made a web crawler which gets all links till the 1st level of page and from them it gets all link and text plus imagelinks and alt. here is whole code: import urllib import re import time from threading import Thread import MySQLdb import mechanize import readability from bs4 import BeautifulSoup from readability.readability import Document import urlparse url = ["http://sparkbrowser.com"] i=0 while i<len(url): counterArray = [0] levelLinks = [] linkText = ["homepage"] levelLinks = [] def scraper(root,steps): urls = [root] visited = [root] counter = 0 while counter < steps: step_url =

Where should put robots.txt? domainname.com/robots.txt or domainname/public_html/robots.txt I placed the file in domainname.com/robots.txt , but it's not opening when I type this in browser. alt text http://shup.com/Shup/358900/11056202047-My-Desktop.png Where the file goes in your filesystem depends on what host you're using, so it's hard for us to give a specific answer about that. The best description is: put it wherever the index.html (or index.php or whatever) file is that represents your homepage. If that's domainname/public_html/index.html , for example, put it in domainname/public_html

I have a robots.txt that is not static but generated dynamically. My problem is creating a route from root/robots.txt to my controller action. This works : routes.MapRoute( name: "Robots", url: "robots", defaults: new { controller = "Home", action = "Robots" }); This doesn't work : routes.MapRoute( name: "Robots", url: "robots.txt", /* this is the only thing I've changed */ defaults: new { controller = "Home", action = "Robots" }); The ".txt" causes ASP to barf apparently You need to add the following to your web.config file to allow the route with a file extension to execute. <?xml version="1

Does Facebook's webcrawling bots respect the Crawl-delay: directive in robots.txt files? We don't have a crawler. We have a scraper that scrapes meta data on pages that have like buttons/are shared on FB. No, it doesn't respect robots.txt Contrary to other answers here, facebookexternalhit behaves like the meanest of crawlers. Whether it got the urls it requests from crawling or from like buttons doesn't matter so much when it goes through every one of those at an insane rate. We sometimes get several hundred hits per second as it goes through almost every url on our site. It kills our servers