Implementing a function that parses a string using bison and flex

Question

I have implemented a parser and scanner using bison and flex. At first it read input from stdin and later I modified to read from a string. There are several posts on stackoverflow on this topic. Specifically, the at flex manual describing the use of multiple buffers and the use of in-memory string rather than a file as input helped.

So, I got the parser/lexer to work for a single string as in:

%{
//Bunch of includes, typedefs, etc..
extern char * yytext;
void yyerror(char *);
int yylex();
typedef struct yy_buffer_state * YY_BUFFER_STATE;
extern int yyparse();
extern int yylex_destroy(void);
extern YY_BUFFER_STATE yy_scan_string(char * str);
extern void yy_delete_buffer(YY_BUFFER_STATE buffer);
...

%%
int main(int argc, char **argv) {

  char *string = "abcd\n\0";
  YY_BUFFER_STATE buffer = yy_scan_buffer(string, sizeof(string));
  yyparse();

}

This was the simple part. The grammar/lex rules are a bit lengthy to reproduce here. I want to implement a function that will be called repeatedly with a string param. Since the grammar/lex rules can take care of the output, I can handle it. Problem is to have a mechanism to implement a classic function to which is passed a string to parse and analyse. So, I tried this to test...

int main(int argc, char **argv) {
  int i=10;
  char *string;
  string = malloc(16);
  while (1) {

     sprintf(string, "someString to be parsed...%d\n\0", i++);
     YY_BUFFER_STATE buffer = yy_scan_buffer(string, sizeof(string));   
     yyparse();
     yylex_destroy();
     printf("%d\n", i); 

   }
}

It fails, after about 10000, with this error:

 fatal flex scanner internal error--end of buffer missed

I am not looking at any multi-threaded environment. It will be a single-threaded process with strings coming from a socket to be parsed. As a string arrives, I call a function to parse it. I am unable to find any reference on implementing such functionality with Bison and Flex.

I tried different approaches but with no luck. Any simple approach to handle this?

You probably have a buffer overrun somewhere in your code that normally is not an issue, but occasionally corrupts things such that you later get this error. Try running with valgrind to see if it tells you anything useful. — Chris Dodd

rici rici · Accepted Answer · 2019-10-11T00:48:22

string = malloc(16);

Is pretty stingy, considering that you are about to write:

sprintf(string, "someString to be parsed...%d\n\0", ++i);

which is at least 28 bytes if i is a single digit and reaches 32 bytes when i reaches 10000. (That's probably not a coincidence.)

Do yourself a favour and use asprintf if you can. If it's not available, it's easy enough to write it using snprintf, or you can just usesnprintf with a much larger buffer. (No need to dynamically allocate in this case.)

Note that the \0 at the end of the format string is totally pointless. I presume that your intention was to guarantee that there are two NUL terminators, as required by yy_scan_buffer, but the \0 will not be copied into sprintf's output since it's effect is to terminate the format string. (C strings are terminated by a NUL character, remember.)

Also note that sizeof(string) in yy_scan_buffer(string, sizeof(string)); is sizeof(char*), since string is a char*. That's most likely 8 these days, but it could be 4 if you're using a 32-bit environment. In any case, it has no relationship with the number of characters which sprintf wrote. You could use strlen to count the length of string but it would be more efficient to take advantage of the fact that sprintf returns the number of bytes it wrote.

And speaking of return values, you don't check the return value from yy_scan_buffer, which probably was trying to tell you about your error:

If you fail to set up base in this manner (i.e., forget the final two YY_END_OF_BUFFER_CHAR bytes), then yy_scan_buffer() returns a NULL pointer instead of creating a new input buffer.

Implementing a function that parses a string using bison and flex

2 Answers