How to detect duplicates among text documents and return the duplicates' similarity?

Question

I\'m writing a crawler to get content from some website, but the content can duplicated, I want to avoid that. So I need a function can return the same percent between two

Answer 1

In bioinformatics there is a algorithm which should do the job. It is called Needleman-Wunsch and is normally used for global sequence alignment with nucletide sequences.

Using this algorithm you could easily calculate accordance between two strings. You can use my code. But this method only returns the alignment you would have to calculate the accordance yourself.