Returning "non-ints" from yylex

4
votes

I have a generated scanner from flex which the output from is not consumed by yacc or bison. yylex() needs to returns a pointer to a token-like structure memory instead of an int indicating the token-type.

// example definition from foo.l
[A-Za-z_][A-Za-z0-9_]* { return scanner_token(T_IDENTIFIER); }

// example implementation of scanner_token
token *scanner_token(name) {
    token *t = (token *)calloc(1, sizeof(token));
    t->name = name;
    t->lexeme = (char *)calloc(yyleng + 1, 1);
    if (t->lexeme == NULL) {
        perror_exit("calloc");
    }
    memmove(t->lexeme, yytext, yyleng);
    return t;
}

// example invocation of yylex
token *t;
t = (token *)yylex();

Of course, the compilation warns me return makes integer from pointer without a cast.

I read in the flex man pages that YY_DECL controls how the scanning routine is declared:

YY_DECL controls how the scanning routine is declared. By default, it is "int yylex()", or, if prototypes are being used, "int yylex(void)". This definition may be changed by redefining the "YY_DECL" macro.

When I try re-defining YY_DECL, the resulting C file fails to compile.

#undef YY_DECL
#define YY_DECL (token *)yylex()

What is the proper way to accomplish what I am trying to?

2
cflex-lexer

2 Answers

8
votes

The normal syntax would be:

#define YY_DECL token *yylex(void)

This minimal Flex source file shows how:

%{
typedef struct token { int tok; } token;
#define YY_DECL token *yylex(void)
token t;
%}
%%
. { t.tok = 1; return &t; }
%%

It compiles for me.

3
votes

yylex should return an int. Keep this behavior, and wrap a call to it in a helper function. The helper returns your token. Don't mess with YY_DECL.

[A-Za-z_][A-Za-z0-9_]* { return T_IDENTIFIER; }

token *scanner_token() {
    token *t;
    int name;
    if ((name = yylex()) == 0)
        return 0;
    else {
        t = (token *)calloc(1, sizeof(token));
        t->name = name;
        t->lexeme = (char *)calloc(yyleng + 1, 1);
        if (t->lexeme == NULL)
            perror_exit("calloc");
        memmove(t->lexeme, yytext, yyleng);
        return t;
    }
}

token *t = scanner_token();