Parsing python with PLY, how to code the indent and dedent part

Question

I was trying to parse the function definition for the python language with PLY. I am encountering issues related to the indentation. For instance for a for statement, I would like to be able to know when the block ends. I read the python grammar here: http://docs.python.org/2/reference/grammar.html And the grammar for this part is:

for_stmt: 'for' exprlist 'in' testlist ':' suite ['else' ':' suite]
suite: simple_stmt | NEWLINE INDENT stmt+ DEDENT

I don't know how to describe the INDENT and DEDENT tokens with PLY. I was trying something like:

def t_indentation(t):
    r'    |\t'
    #some special treatment for the indentation.

But it seems that PLY consider that regexes with spaces match the empty string and does not build the lexer... Even if I would have managed to have the INDENT token I am not sure about the way to get the DEDENT one...

Is there a way to do that with PLY?

The rules about INDENT and DEDENT in Python are more complex than four spaces or a tab. In other parser generators the problem is solved using semantic actions that determine if the amount of leading space is a valid indent or dedent, or by intervening the tokenizer to inject INDENT or DEDENT tokens. — Apalala
Yeah, I was thinking about doing the token injection with the t_indentation function. The regex was wrong for PLY, r'[ ]{4}|\t' is better. Thanks. — joetde
It seems that with plyplus, there is an easier way to do so: github.com/erezsh/plyplus/blob/master/plyplus/grammars/… — joetde

woshifyz woshifyz · Accepted Answer · 2013-11-26T02:23:49

3

votes

You have to use states to parse INDENT and UNDENT.

example of parsing python like language

Parsing python with PLY, how to code the indent and dedent part

2 Answers