Bison Flex cannot access first token in bison rule section

Question

Flex:

%option noyywrap
%{
    #include "syntax.hpp"
    #include <cstdlib>
    extern "C" int isatty(int);
%}
%%

\#.*                                                {}
\[[A-Za-z_]*\]                                      {yylval.val_char=yytext; return SECTION_NAME;}
[A-Z_a-z[:digit:]]*[A-Z_a-z][A-Z_a-z[:digit:]]*     {yylval.val_char=yytext ;return OPTION_NAME;}
\n                                                  {}
=\".*?\"                                            {yylval.val_char=yytext+1; return VALUE_STRING;}
=[ A-Z_a-z[:digit:]]*[A-Z_a-z][A-Z_a-z[:digit:] ]*  {yylval.val_char=yytext+1; return VALUE_ENUM;}
=[ [:digit:] ]*                                     {yylval.val_int=atoi(yytext+1); return VALUE_INTEGER;}
=[ [:digit:]\.[:digit:]]+                           {yylval.val_float=atof(yytext+1); return VALUE_FLOAT;}


" "                                                 {}
%%

Bison:

%{
    #include <iostream> 
    int yylex();
    int yyerror(const char *p) {return 0;}

%}

     %union  {
            float val_float;
            int val_int;
            char* val_char;
           };   
     %token <val_char> SECTION_NAME
     %token <val_char> OPTION_NAME
     %token <val_int> VALUE_INTEGER
     %token <val_float> VALUE_FLOAT
     %token <val_char> VALUE_STRING
     %token <val_char> VALUE_ENUM

%%
    input
        :input line
        |line
        ;
    line
       :section
       |value
       ;


    section
        :SECTION_NAME {std::cout<<$1; std::cout<<std::endl;}
        ;

    value
        :OPTION_NAME VALUE_INTEGER  {std::cout<<$1, std::cout<<std::endl;}
        |OPTION_NAME VALUE_FLOAT {std::cout<<$1; std::cout<<std::endl;}
        |OPTION_NAME VALUE_ENUM {std::cout<<$1; std::cout<<std::endl;}
        |OPTION_NAME VALUE_STRING {std::cout<<$1;std::cout<<std::endl;}
        ;
%%

I am trying to test small ini configuration file parser. It must parse option=value pairs, where value can be int,float,quoted string or enumaerated constant. This code should give SECTION_NAMEs and OPTION_NAMEs frome file to console output. But it does not. Instead of that it prints whole file. If I change $1 to $2 it works fine and prints SECTION_NAMEs and VALUES. What's wrong with it? How can i get access to OPTION_NAME token value? Thanks in advance.

UPD. Flex part works fine. {printf(yylval.val_char); return OPTION_NAME;} gives right token value to console. Perhaps problem in bison part.

Chris Dodd Chris Dodd · Accepted Answer · 2012-11-09T21:30:16

The problem is that you're returning pointers into yytext for your token semantic values. yytext is the internal buffer flex uses to match tokens, and its only valid until the next call to yylex to read the next token. So your semantic values ($1 etc) all end up being dangling pointers into a buffer that's being overwritten with more token data from later tokens.

You need to copy the string out of yytext to some storage that will be available longer. strdup is very useful for this:

\[[A-Za-z_]*\]                                      {yylval.val_char=strdup(yytext); return SECTION_NAME;}
[A-Z_a-z[:digit:]]*[A-Z_a-z][A-Z_a-z[:digit:]]*     {yylval.val_char=stdrup(yytext); return OPTION_NAME;}

etc

Bison Flex cannot access first token in bison rule section

1 Answers