Flex throws "input in flex scanner failed" after parsing whole file?

Question

I am trying to make a parser in C++ using Flex and Bison. I have seen "input in flex scanner failed" online only in posts which provide a bad file for parsing. But I am getting this error after my whole file is parsed correctly. Is there some bug where flex can't detect EOF properly? These are the relevant definitions:

Bison -

%union {
    char node;
    char gate;
    int index;
}

%token END;
%token <node> NODE;
%token <gate> GATE;
%token <index> INDEX;

%%

atpg:
    body END { cout << "Reading last line." << endl; };
body:
    assignments;
assignments:
    assignments assignment { cout << "Token: assignments - rec." << endl; }
    | assignment         { cout << "Token: assignments - first." << endl; };
assignment:
    outnode gatedec '(' nodelist ')'  {
        cout << "Token: assignment" << endl;
        g_gatelist[last_gate].setOutput(&g_nodelist[out_node]);
        for(int i : t_nodelist)
            g_gatelist[last_gate].addInput(&g_nodelist[i]);
        t_nodelist.clear();
        };
outnode:
    nodedec '='  {
        cout << "Token: outnode" << endl;
        out_node = last_node;
        };
nodedec:
    NODE INDEX  {
        num_nodes ++;
        if ($1 == 'i') input_nodes ++;
        if ($1 == 'o') output_nodes ++;

        new(&g_nodelist[$2]) Node($1, $2);
        last_node = $2;

        cout << g_nodelist[$2] << endl;
        };
gatedec:
    GATE INDEX  {
        num_gates ++;

        createGateFromName(&g_gatelist[$2], $1, $2);
        last_gate = $2;

        cout << "Gate: " << $1 << "-" << $2 << endl; 
        };
nodelist:
    nodedec nodelist { 
        cout << "Token: nodelist - rec" << endl;
        t_nodelist.push_back(last_node);
        }
    | nodedec {
        cout << "Token: nodelist - first" << endl;
        t_nodelist.push_back(last_node);
        };

Flex -

%{
#include "gatelist.tab.h"
#define YY_DECL extern "C" int yylex()
%}

INDEX    [0-9]+
GATE     and|or|not
NODE     n|i|o

%%

{GATE}      { yylval.gate = yytext[0]; return GATE; }

{NODE}      { yylval.node = yytext[0]; return NODE; }

{INDEX}     { yylval.index = atoi(yytext); return INDEX; }

[\(\)=]     { return yytext[0]; }

end         { return END; }

.|\n ;/* ignore */
%%
;

Input -

n3 = and1 ( i0 i1 i2 )
n4 = or2 ( i1 i2 )
n5 = not3 ( n4 )
n6 = and4 ( n5 n4 )
end

Output -

Node(n-3 = 0)
Token: outnode
Gate: a-1
Node(i-0 = 0)
Node(i-1 = 0)
Node(i-2 = 0)
Token: nodelist - first
Token: nodelist - rec
Token: nodelist - rec
Token: assignment
Token: assignments - first.
Node(n-4 = 0)
Token: outnode
Gate: o-2
Node(i-1 = 0)
Node(i-2 = 0)
Token: nodelist - first
Token: nodelist - rec
Token: assignment
Token: assignments - rec.
Node(n-5 = 0)
Token: outnode
Gate: n-3
Node(n-4 = 0)
Token: nodelist - first
Token: assignment
Token: assignments - rec.
Node(n-6 = 0)
Token: outnode
Gate: a-4
Node(n-5 = 0)
Node(n-4 = 0)
Token: nodelist - first
Token: nodelist - rec
Token: assignment
Token: assignments - rec.
Reading last line.
input in flex scanner failed

I did the best I could, but please read about how to prepare a minimal reproducible example. It is impossible to test your code because (1) the actions depend on external functions and (2) the driver and Makefile or build procedure are not included. DO NOT just add definitions of the functions needed, since that would make a non-minimal example even longer. Instead, prepare a version without (unnecessary) actions (In your case, all actions are unnecessary) and include all other definitions used (yywrap and yyerror, for example, although I always prefer %option noyywrap in the flex file to avoid the need.) — rici

rici rici · Accepted Answer · 2018-03-15T13:51:24

Bison always augments the provided grammar by adding a goal rule which matches your provided start rule followed by an end-of-file indicator (token number 0), so that the input is forced to match a complete file. If you want to use your own end marker and terminate the parse when it has been recognised, you should use the YYACCEPT action in the associated action to cause the parse to terminate successfully without reading more tokens.

The error you are getting occurs when a flex scanner receives a read error when it attempts to read data. End of file is not an error, and I/O errors are fairly uncommon. Normally, this error indicates that an attempt was made to continue reading input after an end of file had been reported and yyin closed (or set to an invalid value). In general, you should avoid reading tokens after an end-of-file has been reported; although it often works, it is not fully supported and with certain scanner options it will produce unexpected results.

As a side issue, you will usually find that it is easier to use bison's built-in trace facility than to scatter printf's throughout your code. You will end up adding and removing the printf's, which is a nuisance, and they will never give you as complete a picture of what is going on as a bison trace.

Flex throws "input in flex scanner failed" after parsing whole file?

1 Answers