Document Similarity - Multiple documents ended with same similarity score

问题

I have been working in a business problem where i need to find a similarity of new document with existing one. I have used various approach as below

1.Bag of words + Cosine similarity

2.TFIDF + Cosine similarity

3.Word2Vec + Cosine similarity

None of them worked as expected. But finally i found an approach which works better its Word2vec + Soft cosine similarity

But the new challenge is i ended up with multiple documents with same similarity score. Most of them are relevant but few of them even though having some semantically similar words they are different

Please suggest how to over come this issue

回答1:

If the objective is to identify semantic similarity, the following code sourced from here helps.

#invoke libraries
from nltk import pos_tag, word_tokenize
from nltk.corpus import wordnet as wn

#Build functions 
def ptb_to_wn(tag):    
    if tag.startswith('N'):
        return 'n' 
    if tag.startswith('V'):
        return 'v' 
    if tag.startswith('J'):
        return 'a' 
    if tag.startswith('R'):
        return 'r' 
    return None


def tagged_to_synset(word, tag):
    wn_tag = ptb_to_wn(tag)
    if wn_tag is None:
        return None 
    try:
        return wn.synsets(word, wn_tag)[0]
    except:
        return None


def sentence_similarity(s1, s2):    
    s1 = pos_tag(word_tokenize(s1))
    s2 = pos_tag(word_tokenize(s2)) 

    synsets1 = [tagged_to_synset(*tagged_word) for tagged_word in s1]
    synsets2 = [tagged_to_synset(*tagged_word) for tagged_word in s2]

    #suppress "none"
    synsets1 = [ss for ss in synsets1 if ss]
    synsets2 = [ss for ss in synsets2 if ss]

    score, count = 0.0, 0

    for synset in synsets1:
        best_score = max([synset.path_similarity(ss) for ss in synsets2])
        if best_score is not None:
            score += best_score
            count += 1

    # Average the values
    score /= count
    return score

#Build function to compute the symmetric sentence similarity
def symSentSim(s1, s2):
    sss_score = (sentence_similarity(s1, s2) + sentence_similarity(s2,s1)) / 2
    return (sss_score)

#Example
s1 = 'We rented a vehicle to drive to Goa'
s2 = 'The car broke down on our jouney'            

s1tos2 = symSentSim(s1, s2)

print(s1tos2)
#0.155753968254

来源：https://stackoverflow.com/questions/61258035/document-similarity-multiple-documents-ended-with-same-similarity-score

标签

nlp