How to detect duplicates among text documents and return the duplicates' similarity?

Question

I\'m writing a crawler to get content from some website, but the content can duplicated, I want to avoid that. So I need a function can return the same percent between two

Answer 1

A good algorithm for comparing two text is tf-idf. It will give similarity between two documents.

1. calculate tf-idf for the document
2. calculate cosine similarity for two given text
3. the cosine similarity will indicate match between two documents.

This is a very good tutorial for calculating tf-idf and cosine similarity in Java. It would be simple to extend it to C#.