Document Similarity - Multiple documents ended with same similarity score

Question

I have been working in a business problem where i need to find a similarity of new document with existing one. I have used various approach as below

1.Bag of words + Cosine similarity

2.TFIDF + Cosine similarity

3.Word2Vec + Cosine similarity

None of them worked as expected. But finally i found an approach which works better its Word2vec + Soft cosine similarity

But the new challenge is i ended up with multiple documents with same similarity score. Most of them are relevant but few of them even though having some semantically similar words they are different

Please suggest how to over come this issue

Could you please share a sample of about 10 sentences. If this sample has some sentences with acceptable similarity score as well as unacceptable ones, then one could investigate the problem better. — SidharthMacherla
You can try sentence-transformers embedding + cosine similarity as an alternative. — arpitrathi
@SidharthMacherla Due to security policy i couldnot share that info but i try to mimic the scenario...I am having n number of historical docs with different tags or calsses (login issue, error message etc) i just want to find similarity of new document with existing. Now assume i am getting a doc like "Previously i was unable to connect which is resolved but now i am getting error message" due to the key words "unable to connect" and " error message" it gives same similarity score for both class documents eventhough its an error message realted documemt..I hope i have did my best in mimicing. — Sundara Kesavan

SidharthMacherla SidharthMacherla · Accepted Answer · 2020-04-19T05:48:04

If the objective is to identify semantic similarity, the following code sourced from here helps.

#invoke libraries
from nltk import pos_tag, word_tokenize
from nltk.corpus import wordnet as wn

#Build functions 
def ptb_to_wn(tag):    
    if tag.startswith('N'):
        return 'n' 
    if tag.startswith('V'):
        return 'v' 
    if tag.startswith('J'):
        return 'a' 
    if tag.startswith('R'):
        return 'r' 
    return None


def tagged_to_synset(word, tag):
    wn_tag = ptb_to_wn(tag)
    if wn_tag is None:
        return None 
    try:
        return wn.synsets(word, wn_tag)[0]
    except:
        return None


def sentence_similarity(s1, s2):    
    s1 = pos_tag(word_tokenize(s1))
    s2 = pos_tag(word_tokenize(s2)) 

    synsets1 = [tagged_to_synset(*tagged_word) for tagged_word in s1]
    synsets2 = [tagged_to_synset(*tagged_word) for tagged_word in s2]

    #suppress "none"
    synsets1 = [ss for ss in synsets1 if ss]
    synsets2 = [ss for ss in synsets2 if ss]

    score, count = 0.0, 0

    for synset in synsets1:
        best_score = max([synset.path_similarity(ss) for ss in synsets2])
        if best_score is not None:
            score += best_score
            count += 1

    # Average the values
    score /= count
    return score

#Build function to compute the symmetric sentence similarity
def symSentSim(s1, s2):
    sss_score = (sentence_similarity(s1, s2) + sentence_similarity(s2,s1)) / 2
    return (sss_score)

#Example
s1 = 'We rented a vehicle to drive to Goa'
s2 = 'The car broke down on our jouney'            

s1tos2 = symSentSim(s1, s2)

print(s1tos2)
#0.155753968254

Document Similarity - Multiple documents ended with same similarity score

1 Answers