recursive grep using python

Question

I am new to python and trying to learn. I am trying to implement a simple recursive grep using python for processing and here is what I came to so far.

p = subprocess.Popen('find . -name [ch]', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  for line in p.stdout.readlines():
    q = subprocess.Popen('grep searchstring %s', line, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    print q.stdout.readlines()

Can some one pls tell me how to fix this to do what it is supposed to?

Answer 1

You should use the os.walk function for going through your files. Use string methods or regex for filtering out the results. Check http://docs.python.org/library/os.html for informations about how to use os.walk.

import os
import re

def findfiles(path, regex):
    regObj = re.compile(regex)
    res = []
    for root, dirs, fnames in os.walk(path):
        for fname in fnames:
            if regObj.match(fname):
                res.append(os.path.join(root, fname))
    return res

print findfiles('.', r'my?(reg|ex)')

Now for the grep part, you can loop over the file with the open function

def grep(filepath, regex):
    regObj = re.compile(regex)
    res = []
    with open(filepath) as f:
        for line in f:
            if regObj.match(line):
                res.append(line)
    return res

If you want to get the line numbers, you may want to look into the enumerate function.

edited to add the grep function

Answer 2

You can use python-textops3 :

Example, to grep all 'import' in all .py files from current directory :

from textops import *

print('\n'.join(('.' | find('*.py') | cat() | grep('import')))) 

It is pure python, no need to fork a process.

Answer 3

p = subprocess.Popen('find . -name [ch]', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  for line in p.stdout.readlines():
    q = subprocess.Popen('grep searchstring %s', line, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    print q.stdout.readlines()

1. The indentation on line 2 will Exception, the for needs to be aligned with the p above
2. 'grep searchstring %s', line will not do the string replacement, you need to replace the , with %

With those changes and real search values, it works on my OS X box. Final script:

import subprocess
p = subprocess.Popen('find . -name *.py', shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
for line in p.stdout.readlines():
    print line
    q = subprocess.Popen('grep import %s' % line, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    print q.stdout.readlines()

Answer 4

Maybe an example can help you, the command find . -print | grep "python" is equivalent to this:

import subprocess

pc1 = subprocess.Popen('find . -print', stdout=subprocess.PIPE, shell=True)
pc2 = subprocess.Popen('grep "python"', stdin=pc1.stdout, shell=True,
                       stdout=subprocess.PIPE)

print pc2.communicate()[0]