Sentiment analysis using bigrams

Question

So I have some reviews that I am trying to classify into positive or negative. I am trying to use NLTK and Stanford coreNLP to do so. I am able to do it on unigrams but its not working for bigrams. I tried the following for bigrams

def classifySentence(sen):
  wn_lem = WordNetLemmatizer()
  pos = 0
  neg = 0
  stop_words = set(stopwords.words('english'))
  filtered_review = [token for token in nltk.word_tokenize(sen) if not token in stop_words]


  for token in nltk.bigrams(filtered_review):
      #lemma = wn_lem.lemmatize(token)
      # print("lemma="+token)
      if len(wn.synsets(token))>0:
          synset = wn.synsets(token)[0]
          #print("synset.name="+synset.name())

          sent = swn.senti_synset(synset.name())

          #print("Sentiment of "+token+" "+str(sent))

          pos = pos + sent.pos_score()
          neg = neg + sent.neg_score()
          # print (token + "(pos_score): " + str(pos) +"\n")
          # print (token + "(neg_score): " + str(neg) +"\n")
  #print (filtered_review)
  JoinedTokens = ' '.join(wo for wo in filtered_review)
  return [JoinedTokens, pos, neg]

I was wondering if someone could suggest me ways to do this. I would like to use NLTK or can also use stanfordcoreNLP. I am also open to using other python packages but just need some guidance I have written some code for using it but it didn't work either. The code I wrote

def StanfordBigrams():
  nlp = StanfordCoreNLP('http://localhost:9000')
  operations = {'annotators': 'tokenize,lemma,pos,sentiment', 'outputFormat': 'json'}
  string = "not bad"
  tok = nltk.word_tokenize(string)
  bigrams = nltk.bigrams(tok)
  res = nlp.annotate(str(bigrams),operations)
  for s in res["sentences"]: 
          for token in s["tokens"]:
              print("Sentiment: "+str(s["sentiment"])+"SentimentValue: "+str(s["sentimentValue"]))
              print (token)

I would be grateful if someone could point me in the right direction.

Post your errors and stack traces for when it's 'not working'. — John R

Gabor Angeli Gabor Angeli · Accepted Answer · 2018-02-26T23:26:01

Are you training a sentiment classifier, or just trying to use one? Technically, I suspect your error is in wn.synset(bigram) -- I doubt the thing returned from nltk.bigrams is a word that can be passed into WordNet.

But, more importantly, you probably want to pass your whole sentence into a sentiment classifier -- bigrams aren't going to have sentiments annotated on them in things like SentiWordNet, and the trained sentiment classifiers are going to have a much easier time on sentences than they are on short snippets. You should be able to get sentiment for some of the bigrams in the sentence from Stanford's sentiment tree (vs just the sentiment value at the root). See the sentimentTree field on the JSON output from the CoreNLP server.

Sentiment analysis using bigrams

1 Answers