robots.txt

I want know which programming language provides good number of libraries to program a web bot? Something like crawling a web page for data. Say I want fetch weather for weather.yahoo.com website. Also will the answer be same for a AI desktop bot? Here is how you could do it in Python: from urllib2 import urlopen from BeautifulSoup import BeautifulSoup soup=BeautifulSoup(urlopen("http://weather.yahoo.com/").read()) for x in soup.find(attrs={"id":"myLocContainer"}).findAll("li"): print x.a["title"], x.em.contents Prints: Full forecast for Chicago, Illinois, United States (Haze) [u'35...47 °F']

I have a serious question. Is it ever ethical to ignore the presence of a robots.txt file on a website? These are some of the considerations I've got in mind: If someone puts a web site up they're expecting some visits. Granted, web crawlers are using bandwidth without clicking on ads that may support the site but the site owner is putting their site on the web, right, so how reasonable is it for them to expect that they'll never get visited by a bot? Some sites apparently use a robots.txt exactly in order to keep their site from being crawled by Google or some other utility that might grab

When a crawler reads the User-Agent line of a robots.txt file, does it attempt to match it exactly to its own User-Agent or does it attempt to match it as a substring of its User-Agent? Everything I have read does not explicitly answer this question. According to another StackOverflow thread it is an exact match. However, the RFC draft makes me believe that it is a substring match. For example, User-Agent: Google will match "Googlebot" and "Googlebot-News". Here is the relevant quotation from the RFC: The robot must obey the first record in /robots.txt that contains a User-Agent line whose

I want to add nofollow and noindex to my site whilst it's being built. The client has request I use these rules. I am aware of <meta name="robots" content="noindex,nofollow"> But I only have access to the robots.txt file. Does anyone know the correct format I can use to apply noindex, nofollow rules via the robots.txt file? noindex and nofollow means you do not want your site to crawl in search engine. so simply put code in robots.txt User-agent: * Disallow: / it means noindex and nofollow. There is a non-standard Noindex field, which Google (and likely no other consumer) supported as

how to disallow all dynamic urls in robots.txt Disallow: /?q=admin/ Disallow: /?q=aggregator/ Disallow: /?q=comment/reply/ Disallow: /?q=contact/ Disallow: /?q=logout/ Disallow: /?q=node/add/ Disallow: /?q=search/ Disallow: /?q=user/password/ Disallow: /?q=user/register/ Disallow: /?q=user/login/ i want to disallow all things that start with /?q= The answer to your question is to use Disallow: /?q= The best (currently accessible) source on robots.txt I could find is on Wikipedia . (The supposedly definitive source is http://www.robotstxt.org , but site is down at the moment.) According to the

I'm currently working on an application built with Express (Node.js) and I want to know what is the smartest way to handle different robots.txt for different environments (development, production). This is what I have right now but I'm not convinced by the solution, I think it is dirty: app.get '/robots.txt', (req, res) -> res.set 'Content-Type', 'text/plain' if app.settings.env == 'production' res.send 'User-agent: *\nDisallow: /signin\nDisallow: /signup\nDisallow: /signout\nSitemap: /sitemap.xml' else res.send 'User-agent: *\nDisallow: /' (NB: it is CoffeeScript) There should be a better way