Why the return value for more than a token?

Question

I want to return DOLLARID($foo) and DOTID(.foo),so I wirte tow rule(A code snippet of my lex file):

ID  ([_a-zA-Z]+[a-zA-Z0-9_\-]*)
DOLLAR ("$"|("$!"))
DOT "."
%x DIRECTIVE REFERENCE
%%

[^#$]*?/"$" {BEGIN REFERENCE;yylval.string = yytext;printf("==========begin reference flex content===content:%s=====\n",yytext);return CONTENT;}
[^$#]*?/"#" {BEGIN DIRECTIVE;yylval.string = yytext; return CONTENT;}
<REFERENCE,DIRECTIVE>{DOLLAR}{ID} {yylval.string = yytext;printf("==========flex    content===ID:%s=====\n",yytext);return DOLLARID;}
<REFERENCE,DIRECTIVE>{DOT}{ID} {yylval.string = yytext;printf("==========flex content===DOTID:%s=====\n",yytext);return DOTID;}

A code snippet of my yacc file:

set:SET PARENTHESIS reference EQUAL expression CLOSE_PARENTHESIS { $$ = set_directive($3,$5); }
;
reference: DOLLARID {printf("reference ---Id,key:%s\n",$1);$$ = reference($1);}
|DOLLARID DOTID {printf("reference ---dotId\n");$$ = reference($2);}
;

I write a test file test.vm

#set($arr = [1..5])
#set($hell = "sinory")
$hell
$arr

when I run it,a part of the result is:

line 1 is printed by lexer, it's right

line 2 is printed by bison, it's more than two characters(" =")

Because of flex needs more than a token？

I don't know why?please help me to fix it.

Chris Dodd Chris Dodd · Accepted Answer · 2013-04-15T06:23:03

The problem is that yytext is only valid for a single token and will get overwritten or otherwise modified by the next token read. So returning a pointer into it will generally not work -- it will have the token text for a little while but will later change out from under you. You need to make a copy of the string in yytext if you want to actually make use of its value in the parser.

Change your lexer code to use yylval.string = strdup(yytext); and things will work better (though then you need to worry about freeing the strings to avoid leaking memory).

Why the return value for more than a token?

1 Answers