iterating over file object in Python does not work, but readlines() does but is inefficient

3
votes

In the following code, if I use: 

for line in fin:

It only executes for 'a'

But if I use:

wordlist = fin.readlines()
for line in wordlist:

Then it executes for a thru z.

But readlines() reads the whole file at once, which I don't want.

How to avoid this?

def avoids():
    alphabet = 'abcdefghijklmnopqrstuvwxyz'
    num_words = {}

    fin = open('words.txt')

    for char in alphabet:
      num_words[char] = 0
      for line in fin:
        not_found = True
        word = line.strip()
        if word.lower().find(char.lower()) != -1:
          num_words[char] += 1
    fin.close()
    return num_words
3
python
You have two loops - in any case you will need entire file - my suggestion is read it all. - Artsiom Rudzenka
@Greg Lafrance: Further to Artsiom Rudzenka's my suggestion is read it all.... my suggestion is: read it all ONCE!!! by inverting your loop logic to foreach line in file: foreach letter in alphabet (instead of foreach letter in alphabet: foreach line in file... in general I/O should allways be the outter of two loops, becuase I/O is much much much much much slower than memory access. - corlettk

3 Answers

8
votes

the syntax for line in fin can only be used once. After you do that, you've exhausted the file and you can't read it again unless you "reset the file pointer" by fin.seek(0). Conversely, fin.readlines() will give you a list which you can iterate over and over again.

I think a simple refactor with Counter (python2.7+) could save you this headache:

from collections import Counter
with open('file') as fin:
    result = Counter()
    for line in fin:
        result += Counter(set(line.strip().lower()))

which will count the number of words in your file (1 word per line) that contain a particular character (which is what your original code does I believe ... Please correct me if I'm wrong)

You could also do this easily with a defaultdict (python2.5+):

from collections import defaultdict
with open('file') as fin:
    result = defaultdict(int)
    for line in fin:
        chars = set(line.strip().lower())
        for c in chars:
            result[c] += 1

And finally, kicking it old-school -- I don't even know when setdefault was introduced...:

fin = open('file')
result = dict()
for line in fin:
    chars = set(line.strip().lower())
    for c in chars:
        result[c] = result.setdefault(c,0) + 1

fin.close()
5
votes

You have three options:

  1. Read in the whole file anyway.
  2. Seek back to the beginning of the file before attempting to iterate over it again.
  3. Rearchitect your code so that it doesn't need to iterate over the file more than once.
0
votes

Try:

from collections import defaultdict
from itertools import product

def avoids():
    alphabet = 'abcdefghijklmnopqrstuvwxyz'

    num_words = defaultdict(int)

    with open('words.txt') as fin:
        words = [x.strip() for x in fin.readlines() if x.strip()]

    for ch, word in product(alphabet, words):
        if ch not in word:
             continue
        num_words[ch] += 1

    return num_words