search-engine

I don't want the search engines to index my imprint page. How could I do that? You need a simple robots.txt file. Basically, it's a text file that tells search engines not to index particular pages. You don't need to include it in the header of your page; as long as it's in the root directory of your website it will be picked up by crawlers. Create it in the root folder of your website and put the following text in: User-Agent: * Disallow: /imprint-page.htm Note that you'd replace imprint-page.html in the example with the actual name of the page (or the directory) that you wish to keep from

This question already has an answer here: crawler vs scraper 4 answers Is there a difference between Crawling and Web-scraping? If there's a difference, what's the best method to use in order to collect some web data to supply a database for later use in a customised search engine? Ben Crawling would be essentially what Google, Yahoo, MSN, etc. do, looking for ANY information. Scraping is generally targeted at certain websites, for specfic data, e.g. for price comparison, so are coded quite differently. Usually a scraper will be bespoke to the websites it is supposed to be scraping, and would

I'm building a Django project that needs search functionality, and until there's a django.contrib.search , I have to choose a search app. So, which is the best? By "best" I mean... easy to install / set up has a Django- or at least Python-friendly API can perform reasonably complex searches Here are some apps I've heard of, please suggest others if you know of any: djangosearch django-sphinx I'd also like to avoid using a third-party search engine (like Google SiteSearch), because some of the data I'd like to index is for site members only and should not be public. kpw Check out Haystack

I'm new to search engines and web crawlers. Now I want to store all the original pages in a particular web site as html files, but with Apache Nutch I can only get the binary database files. How do I get the original html files with Nutch? Does Nutch support it? If not, what other tools can I use to achieve my goal.(The tools that support distributed crawling are better.) Well, nutch will write the crawled data in binary form so if if you want that to be saved in html format, you will have to modify the code. (this will be painful if you are new to nutch). If you want quick and easy solution