Flex regular expression literal char

Question

I'm having some trouble using flex to setup a regular expression to match C-like literal chars.

I need to match the literal chars that are correct according to syntax and the incorrect ones, such has the unterminated char literals.

2 rules, one for the correct one's and one for the unterminated.

chrlit          (\')([^\\\'\n]|(\\.))(\')
untermchrlit    (\')([\\|\']|(.))*

I need help with the regular expressions because they're not working as I need them. Here are some examples how they should work:

'          -> unterminated char constant
'/'        -> CHRLIT('/')
'('        -> CHRLIT('(')
'a"b"c"de  -> unterminated char constant
'abc       -> unterminated char constant
'abc\      -> unterminated char constant
'\\'       -> CHRLIT('\\')
';'        -> CHRLIT(';')
''         -> unterminated char constant
'a'        -> CHRLIT('a')
'\'        -> unterminated char constant 
'\;'       -> CHRLIT('\;')
'\\\'      -> unterminated char constant
'\\\       -> unterminated char constant    
'\/'       -> CHRLIT('\/')
'a\'       -> unterminated char constant
'\\        -> unterminated char constant
'\t'       -> CHRLIT('\t')

@MikeM it's a char literal so only one char. I have another rule to match multi char constants so they give an error but that one I think it's not relevant to the question ;) — andrepcg
@MikeM my untermchrlit is failing in '\', 'a\', ''' and some other cases. it's giving me weird results.. — andrepcg

rici rici · Accepted Answer · 2013-03-22T00:38:56

The problem is that your pattern for unterminated character literals will also match a terminated one as well as any following characters, unless the character literal is at the end of a line. Rather than try to precisely match an unterminated character literal, you could make life simpler on yourself like this, which falls back to untermchrlit if it encounters an ' which is not the start of a chrlit. (So it must be unterminated, if chrlit matches all possible terminated literals.) (I've also taken the liberty of removing all of the excess parentheses and backslashes from your regular expressions, which makes them a bit less noisy to read.)

chrlit          '([^'\\\n]|\\.)'
untermchrlit    '

The only problem with this solution is that it will continue the scan right after the unterminated ', which is highly likely to create artificial errors, particularly in the case that there really was a matching ', as in 'too long'. Here you really want to continue the lexical scan after the second ' (and, indeed, you'd probably want to flag this as an overly-long character literal rather than an unterminated one). To handle that case, you would need a more sophisticated set of patterns. Here's on possibility:

/* As before */
chrlit          '([^'\\\n]|\\.)'
/* Also as before, a catch-all case. */
untermchrlit    '
/* Try to match single-quoted strings which are too short or too long */
emptychrlit     ''
/* The action for this regex *must* come after the action for chrlit */
longchrlit      '([^'\\\n]|\\.)+'

I should note that longchrlit here also matches everything that chrlit matches, but unlike the pattern in the OP, it does not match any more characters. It's important that the actions be ordered as indicated by the comment, so that correct literals will be matche by chrlit. (If you get the order wrong, flex should issue a warning, though.)

Remember that Flex always matches the longest match, but if more than one rule matches exactly the same token, Flex chooses the first action.

By the way, at least in C, the following is a valid character literal:

'a\
'

That's because \ followed immediately by a newline is completely removed from the input, so the second ' is lexed as though it were immediately following the a.

Flex regular expression literal char

1 Answers