Recommendations for a spidering tool to use with Lucene or Solr? [closed]

问题

As it currently stands, this question is not a good fit for our Q&A format. We expect answers to be supported by facts, references, or expertise, but this question will likely solicit debate, arguments, polling, or extended discussion. If you feel that this question can be improved and possibly reopened, visit the help center for guidance.

Closed 6 years ago.

What is a good crawler (spider) to use against HTML and XML documents (local or web-based) and that works well in the Lucene / Solr solution space? Could be Java-based but does not have to be.

回答1:

In my opinion, this is a pretty significant hole which is keeping down the widespread adoption of Solr. The new DataImportHandler is a good first step to import structured data, but there is not a good document ingestion pipeline for Solr. Nutch does work, but the integration between Nutch crawler and Solr is somewhat clumsy.
I've tried every open-source crawler that I can find, and none of them integrates out-of-the-box with Solr.
Keep an eye on OpenPipeline and Apache Tika.

回答2:

I've tried nutch, but it was very difficult to integrate with Solr. I would take a look at Heritrix. It has an extensive plugin system to make it easy to integrate with Solr, and it is much much faster at crawling. It makes extensive use of threads to speed up the process.

回答3:

I suggest you to check out Nutch to get some inspiration:

Nutch is open source web-search software. It builds on Lucene Java, adding web-specifics, such as a crawler, a link-graph database, parsers for HTML and other document formats, etc.

回答4:

Also check Apache Droids [http://incubator.apache.org/droids/] -- this hopes not be a simple spider/crawler/worker framework.

It is new and is not yet easy to use off the shelf (it will take some tweeking to get running), but is a good thing to keep your eye on.

回答5:

Nutch might be your closest match, but it's not too flexible.

If you need something more you will have to pretty much hack your own crawler. It's not as bad as it sounds, every language has web libraries, so you just need to connect some task queue manager with HTTP downloader and HTML parser, it's not really that much work. You can most likely get away with a single box, as crawling is mostly bandwidth-intentive, not CPU-intensive.

回答6:

http://arachnode.net

C#, but produces Lucene (Java and C#) consumable index files.

回答7:

Did anyone tried Xapian? It seams much quicker than solr and written in c++.

来源：https://stackoverflow.com/questions/282654/recommendations-for-a-spidering-tool-to-use-with-lucene-or-solr