directory - Iterating through directories with Python

183
votes

I need to iterate through the subdirectories of a given directory and search for files. If I get a file I have to open it and change the content and replace it with my own lines.

I tried this:

import os

rootdir ='C:/Users/sid/Desktop/test'

for subdir, dirs, files in os.walk(rootdir):
    for file in files:
        f=open(file,'r')
        lines=f.readlines()
        f.close()
        f=open(file,'w')
        for line in lines:
            newline = "No you are not"
            f.write(newline)
        f.close()

but I am getting an error. What am I doing wrong?

3
pythondirectory
"An error" - any error in particular? - Daniel Roseman
Please could you explain a little about what you hope to do with the files / directories once you get the walk through them working as intended? Also please provide error details. - ChrisProsser
The error message that im getting is that the file cool.txt is not found. In my test folder i have an other folder called src and in the src folder i have another folder called main, in this folder i have cool.txt - Wolf
can you just write the error in the question? its beyond annoying and unnecessary to have to read through the comments to find it. - Charlie Parker
over a year later I can't believe I'm back requesting for the error to be posted? @Wolf - Charlie Parker

3 Answers

360
votes

The actual walk through the directories works as you have coded it. If you replace the contents of the inner loop with a simple print statement you can see that each file is found:

import os
rootdir = 'C:/Users/sid/Desktop/test'

for subdir, dirs, files in os.walk(rootdir):
    for file in files:
        print os.path.join(subdir, file)

If you still get errors when running the above, please provide the error message.

Updated for Python3

import os
rootdir = 'C:/Users/sid/Desktop/test'

for subdir, dirs, files in os.walk(rootdir):
    for file in files:
        print(os.path.join(subdir, file))
25
votes

Another way of returning all files in subdirectories is to use the pathlib module, introduced in Python 3.4, which provides an object oriented approach to handling filesystem paths (Pathlib is also available on Python 2.7 via the pathlib2 module on PyPi):

from pathlib import Path

rootdir = Path('C:/Users/sid/Desktop/test')
# Return a list of regular files only, not directories
file_list = [f for f in rootdir.glob('**/*') if f.is_file()]

# For absolute paths instead of relative the current dir
file_list = [f for f in rootdir.resolve().glob('**/*') if f.is_file()]

Since Python 3.5, the glob module also supports recursive file finding:

import os
from glob import iglob

rootdir_glob = 'C:/Users/sid/Desktop/test/**/*' # Note the added asterisks
# This will return absolute paths
file_list = [f for f in iglob(rootdir_glob, recursive=True) if os.path.isfile(f)]

The file_list from either of the above approaches can be iterated over without the need for a nested loop:

for f in file_list:
    print(f) # Replace with desired operations
19
votes

From python >= 3.5 onward, you can use **, glob.iglob(path/**, recursive=True) and it seems the most pythonic solution, i.e.:

import glob, os

for filename in glob.iglob('/pardadox-music/**', recursive=True):
    if os.path.isfile(filename): # filter dirs
        print(filename)

Output:

/pardadox-music/modules/her1.mod
/pardadox-music/modules/her2.mod
...

Notes:
1 - glob.iglob

glob.iglob(pathname, recursive=False)

Return an iterator which yields the same values as glob() without actually storing them all simultaneously.

2 - If recursive is True, the pattern '**' will match any files and zero or more directories and subdirectories.

3 - If the directory contains files starting with . they won’t be matched by default. For example, consider a directory containing card.gif and .card.gif:

>>> import glob
>>> glob.glob('*.gif') ['card.gif'] 
>>> glob.glob('.c*')['.card.gif']

4 - You can also use rglob(pattern), which is the same as calling glob() with **/ added in front of the given relative pattern.