How to write a crawler?

Question

I have had thoughts of trying to write a simple crawler that might crawl and produce a list of its findings for our NPO\'s websites and content.

Does anybody have an

Answer 1

I'm using Open search server for my company internal search, try this : http://open-search-server.com its also open soruce.