Catching String Literal Errors using Flex-Lexer

Question

I have managed to effectively match valid string literals in my Flex program, but I would also like to match unterminated string literals and string literals with bad escape sequences.

For example, my string literals are matched using simple regex as such:

\"(\\.|[^\\"])*\"

Then I tried to find where a string literal is started with a " then some text, and then \n. This is incorrect syntax for my lexer, and I'd like to catch and produce an error out.

My current regex I came up with for that is this:

\"(\\.|[^\\"])*\n

Which does correctly catch the error, but then seems to eat up the rest of the tokens, because there's no output after that.

Additionally, I was also looking to have a special case error for when an unterminated string literal had an invalid escape sequence. For example:

"some text \
int abc

So my question boils down to, is there something wrong with my current way of matching string literals that's affecting my ability to catch these errors, or is my pattern matching unnecessarily consuming tokens? It's also possible I have no idea what I'm doing!

Some examples of strings:

"a correct string literal"
"an unterminated string literal
"an unterminated string literal with escape \

All string literals are single-line and follow the form:

"(.*)"\n

In C, a line which ends with a backslash is continued on the next line. That works in string literals, too. (Although there are some corner cases with "splices" because they are handled before any lexical analysis, at least in theory.) An example of an invalid escape sequence would be \Z. — rici

rici rici · Accepted Answer · 2021-09-02T01:11:53

The correct flex pattern for string literals is (see below for escape sequences):

\"(\\(.|\n)|[^\\"\n])*\"

This differs from your pattern in that it allows a newline after an escape character (which is technically a splice, rather than part of the syntax of string literals [Note 1]), and bans newlines otherwise. That has to be done explicitly because [^...] includes a newline unless \n is part of the list of characters to be rejected. Only . implicitly bans newlines.

To match incorrect string literals, you only need the same pattern without the terminating ":

\"(\\(.|\n)|[^\\"\n])*

You don't need to worry about that pattern matching correct string literals, because flex always chooses the longest match, and the match with the terminating quote is guaranteed to be longer.

If you want to be more accurate about escape characters, you would need something like:

\"(\\([abfnrtv'"?\\\n]|[0-7]{1,3}|x[[:xdigit:]]+|u[[:xdigit:]]{4}|U[[:xdigit:]]{8})|[^\\"\n])*\"

You can use the same technique to match errors, but you might want to distinguish between unterminated quote errors and invalid escape errors, which you can do by using two error patterns:

\"(\\([abfnrtv'"?\\\n]|[0-7]{1,3}|x[[:xdigit:]]+|u[[:xdigit:]]{4}|U[[:xdigit:]]{8})|[^\\"\n])*\"   { /* Valid string */ }
\"(\\([abfnrtv'"?\\\n]|[0-7]{1,3}|x[[:xdigit:]]+|u[[:xdigit:]]{4}|U[[:xdigit:]]{8})|[^\\"\n])*/\\  { /* Invalid escape sequence */ }
\"(\\([abfnrtv'"?\\\n]|[0-7]{1,3}|x[[:xdigit:]]+|u[[:xdigit:]]{4}|U[[:xdigit:]]{8})|[^\\"\n])*     { /* Missing terminating quote */ }

Notes

A "splice" is a backslash at the end of a line. You commonly see these only in definitions of long macros, but C allows a splice anywhere: the backslash and the following newline are just deleted from the program text, so a splice can even be placed in the middle of an identifier or multicharacter operator. (But don't do that!)

Using splices to continue strings over multiple lines is not good style; it's better to use string concatenation. But the C standard does allow it.

However, splices are removed before tokenisation starts, which means that you cannot backslash escape a backslash at the end of a line:
```
"This is a string literal which includes a \\
t tab, with a splice in the middle of the escape."
```
Please don't use that in production code, either :-)

Catching String Literal Errors using Flex-Lexer

1 Answers

Notes