Word Frequency HW

Question

Write a program that asks a user for a file name, then reads in the file. The program should then determine how frequently each word in the file is used. The words should be counted regardless of case, for example Spam and spam would both be counted as the same word. You should disregard punctuation. The program should then output the the words and how frequently each word is used. The output should be sorted by the most frequent word to the least frequent word.

Only problem I am having is getting the code to count "The" and "the" as the same thing. The code counts them as different words.

userinput = input("Enter a file to open:")
if len(userinput) < 1 : userinput = 'ran.txt'
f = open(userinput)
di = dict()
for lin in f:
    lin = lin.rstrip()
    wds = lin.split()
    for w in wds:
        di[w] = di.get(w,0) + 1
    lst = list()
    for k,v in di.items():
       newtup = (v, k)
       lst.append(newtup)
lst = sorted(lst, reverse=True)
print(lst)

Need to count "the" and "The" as on single word.

Answer 1

We start by getting the words in a list, updating the list so that all words are in lowercase. You can disregard punctuation by replacing them from the string with an empty character

punctuations = '!"#$%&\'()*+,-./:;<=>?@[\\]^_`{|}~'
s = "I want to count how many Words are there.i Want to Count how Many words are There"

for punc in punctuations:
    s = s.replace(punc,' ')

words = s.split(' ')
words = [word.lower() for word in words]

We then iterate through the list, and update a frequency map.

freq = {}

for word in words:
    if word in freq:
        freq[word] += 1
    else:
        freq[word] = 1
print(freq)
#{'i': 2, 'want': 2, 'to': 2, 'count': 2, 'how': 2, 'many': 2, 
#'words': 2, 'are': #2, 'there': 2}

Answer 2

You can use counter and re like this,

from collections import Counter
import re

sentence = 'Egg ? egg Bird, Goat  afterDoubleSpace\nnewline'

# some punctuations (you can add more here)
punctuationsToBeremoved = ",|\n|\?" 

#to make all of them in lower case
sentence = sentence.lower() 

#to clean up the punctuations
sentence = re.sub(punctuationsToBeremoved, " ", sentence) 

# getting the word list
words = sentence.split()

# printing the frequency of each word
print(Counter(words))