Python NLTK Naive Bayes Classifier

Question

I'm trying to implement NLTK Naive Bayes Classifier on a dataset which has positive and negative categories with a feature extraction function features_all(). When I run the code I get an error on a line in features_all() function.

Code for Naive Bayes:

import nltk
import random
from nltk.corpus import stopwords
import nltk.classify.util
from nltk.corpus.reader import CategorizedPlaintextCorpusReader
import re

from feature_extractors import features_all #function for features extraction

path = "/.../all kom"

reader = CategorizedPlaintextCorpusReader(path,r'.*\.txt',cat_pattern=r'(^\w..)/*')

po=reader.sents(categories=['pos']) #tokenize 
ne=reader.sents(categories=['neg'])

labeled_sentiments = ([(n, 'positive') for n in po] + [(n, 'negative') for n in ne])

size = int(len(labeled_sentiments) * 0.9) #for separating training set in 90:10
random.shuffle(labeled_sentiments)

featuresets = [(features_all(n), sentiment) for (n, sentiment) in labeled_sentiments]
train_set = featuresets[:size]
test_set = featuresets[size:]

#Naive Bayes
classifier = nltk.NaiveBayesClassifier.train(train_set)
#test
print(classifier.classify(features_all('great')))
print(classifier.classify(features_all('bad')))
print('Accuracy for Naive Bayes: ',nltk.classify.accuracy(classifier,   test_set))
print(classifier.show_most_informative_features(15))

features_all() function:

def features_all(dat):

    f_all_dict=open('all_dict.txt','r',encoding='utf-8').read()

    f = literal_eval(f_all_dict)

    result_all = {} 

    for word in f.items():
        result_all = {"{}_{}".format(word, suffix): pol * dat.count(word) for word, (suffix, pol) in f.items()} #here is where I get the error

    if len(f) == len(result_all):
       return result_all
    else:
       return None

And features_all() gives an output like (example):

great_pos:1, bad_neg:1

and all_dict.txt looks like this:

"great":("pos",2),"bad":("neg",2)

I get the error on line result_all = {"{}_{}".format(word, suffix): pol * dat.count(word) for word, (suffix, pol) in f.items()}

Since I don't know exactly what is the error, because when I run the code it doesn't want to finish executing, so I stop execution and here is where it stops, so I'm pretty sure it's on this line. I'm a bit confused I don't know anymore if the problem is in formatting or function input. If someone can help i would appreciate.

Have you tried using this? "{}_{}:{}".format(word, suffix, pol * dat.count(word)) — Ic3fr0g

Ic3fr0g Ic3fr0g · Accepted Answer · 2016-05-31T13:07:01

Pretty sure that you just need to include "{}_{}:{}".format(word, suffix, pol * dat.count(word)) for word, (suffix, pol) in f.items() in the formatted return statement for results_all. A very easy way to check whether your code works is to check whether you are consistently getting the outputs in the format you expect! If you simply did print("{}_{}".format(word, suffix): pol * dat.count(word) for word, (suffix, pol) in f.items()) ,you get an Invalid Syntax error. Keep print statements if you're unsure about code!

Python NLTK Naive Bayes Classifier

1 Answers