Issues with nested list comprehension in Python text analysis

Question

So, very much like is being done in this site's Introduction to Text Analysis with Python, I'm trying to use a given list of positive words to gauge the sentiment of a large set of tweets, but with list comprehension rather than for loops. However, somewhere along the line, something doesn't work, because the final zipped result of the tweets along with what should be the number of positive words in them is mostly incorrect. Here are the relevant lines:

positive = open('positive.txt').read().split()
posTweets = [sum([tweet.count(word) for word in positive]) for tweet in tweets]
posTweetTuples = zip(tweets, posTweets)

Now, my intuition tells me the second line is the offender here, given it's the more complicated line, and it's my first time nesting list comprehension. But this seems like it should work. The idea is that it looks through every word in every tweet to see if it matches a word from positive.txt, then sums up how many times that happens per tweet, and zips that list together with the original list of tweets. This is really inefficient and takes a few seconds to run, but that's not the concern here. I do a good amount of preprocessing, but I don't think it's the issue here. For an example of things being wrong, this is one of my results:

('the 2016 presidential campaign a news event thats hard to miss https//tco/gvuvw8ay6v', 1),

But searching every word in the tweet against positive.txt reveals that the number should be 0. Did I just mess up the nesting?

Overall, I can tell that something is wrong because I've also separated it out by every individual word instead of every individual tweet, and comparing that list of words against the positive words list gives me a result of 1105, yet with the provided code above, the total sum calculated that way is 3326. Given that the method for calculating against a bag of words is much simpler (and the example given above), I assume that the 1105 number is correct, while the 3326 is the incorrect one.

Answer 1

Running the tutorial count to get a sum of positive words in the tweets gives me 753 hits, not 1105.

Your list comprehensions are correct. However, the tutorial splits the tweets by whitespace, then counts whole words:

>>> for tweet in tweets_list:
...     positive_counter=0
...     tweet_processed=tweet.lower()
...     for p in list(punctuation):
...         tweet_processed=tweet_processed.replace(p,'')
...         words=tweet_processed.split(' ')
...     for word in words:
...         if word in positive_words:
...             print word
...             positive_counter=positive_counter+1
...      print positive_counter/len(words)

You count partial words. Take the following tweet:

Romney and Obama agree that Augusta National should allow women to be members? Unthinkable...and bad news for the green coats.

The positive words able and thinkable are in that tweet, except they are only part of the word Unthinkable. They should not have been counted.

You could use a regex here to catch word boundaries:

import re

positive_pattern = re.compile(r'\b(?:{})\b'.format('|'.join(map(re.escape, positive))))
posTweets = [len(positive_pattern.findall(tweet)) for tweet in tweets]

This returns fewer hits, only 583, because you still have punctuation in your tweets. Rather than use the loop with str.replace() for each punctuation character, use str.translate():

positive_pattern = re.compile(r'\b(?:{})\b'.format('|'.join(map(re.escape, positive))))
no_punctuation = dict.fromkeys(map(ord, punctuation))
posTweets = [len(positive_pattern.findall(tweet.translate(no_punctuation)) for tweet in tweets]

This gets you the same result as the tutorial code, now the count is up to 753.