How does one parse a .lua file with Python and pull out the require statements?

Question

I am not very good at parsing files but have something I would like to accomplish. The following is a snippet of a .lua script that has some require statements. I would like to use Python to parse this .lua file and pull the 'require' statements out.

For example, here are the require statements:

require "common.acme_1"
require "common.acme_2"
require "acme_3"
require "common.core.acme_4"

From the example above I would then like to split the directory from the required file. In the example 'require "common.acme_1"' the directory would be common and the required file would be acme_1. I would then just add the .lua extention to acme_1. I need this information so I can validate if the file exists on the file system (which I know how to do) and then against luac (compiler) to make sure it is a valid lua file (which I also know how to do).

I simply need help pulling these require statements out using Python and splitting the directory name from the filename.

Answer 1

You can do this with built in string methods, but since the parsing is a little bit complicated (paths can be multi-part) the simplest solution might be to use regex. If you're using regex, you can do the parsing and splitting using groups:

import re


data = \
'''
require "common.acme_1"
require "common.acme_2"
require "acme_3"
require "common.core.acme_4"
'''


finds = re.findall(r'require\s+"(([^."]+\.)*)?([^."]+)"', data, re.MULTILINE)

print [dict(path=x[0].rstrip('.'),file=x[2]) for x in finds]

The first group is the path (including the trailing .), the second group is the inner group needed for matching repeated path parts (discarded), and the third group is the file name. If there is no path you get path=''.

Output:

[{'path': 'common', 'file': 'acme_1'}, {'path': 'common', 'file': 'acme_2'}, {'path': '', 'file': 'acme_3'}, {'path': 'common.core', 'file': 'acme_4'}]

Answer 2

Here ya go!

import sys
import os.path
if len(sys.argv) != 2:
    print "Usage:", sys.argv[0], "<inputfile.lua>"
    exit()
f = open(sys.argv[1], "r")
lines = f.readlines()
f.close()
for line in lines:
    if line.startswith("require "):
        path = line.replace('require "', '').replace('"', '').replace("\n", '').replace(".", "/") + ".lua"
        fName = os.path.basename(path)
        path = path.replace(fName, "")
        print "File: " + fName
        print "Directory: " + path
        #do what you want to each file & path here

Answer 3

Here's a crazy one-liner, not sure if this was exactly what you wanted and most certainly not the most optimal one...

In [270]: import re

In [271]: [[s[::-1] for s in rec[::-1].split(".", 1)][::-1] for rec in re.findall(r"require \"([^\"]*)", text)]
Out[271]: 
[['common', 'acme_1'],
 ['common', 'acme_2'],
 ['acme_3'],
 ['common.core', 'acme_4']]

Answer 4

This is straight forward

One liners are great but they take too much effort to understand early and this is not a job for using regular expressions in my opinion

mylines = [line.split('require')[-1] for line in open(mylua.lua).readlines() if line.startswith('require')]

paths = []
for line in mylines:
    if 'common.' in line:
        paths.append('common, line.split('common.')[-1]
    else:
        paths.append('',line)

Answer 5

You could use finditer:

lua='''
require "common.acme_1"
require "common.acme_2"
require "acme_3"
require 'common.core.acme_4'
'''

import re
print [m.group(2) for m in re.finditer(r'^require\s+(\'|")([^\'"]+)(\1)', lua, re.S | re.M)]
# ['common.acme_1', 'common.acme_2', 'acme_3', 'common.core.acme_4']

Then just split on the '.' to split into paths:

for e in [m.group(2) for m in re.finditer(r'^require\s+(\'|")([^\'"]+)(\1)', lua, re.S | re.M)]:
    parts=e.split('.')
    if parts[:-1]:
        print '/'.join(parts[:-1]), parts[-1]
    else:
        print parts[0]  

Prints:

common acme_1
common acme_2
acme_3
common/core acme_4

Answer 6

file = '/path/to/test.lua'


def parse():
    with open(file, 'r') as f:
        requires = [line.split()[1].strip('"') for line in f.readlines() if line.startswith('require ')]

    for r in requires:
        filename = r.replace('.', '/') + '.lua'
        print(filename)

The with statement opens the file in question. The next line creates a list of all lines that start with 'require ' and splits them, ignoring the 'require' and grabbing only the last part and strips off the double quotes. Then go though the list and replace the dots with slashes and appends '.lua'. The print statement shows the results.