Unrecognized rule error

Question

    %option lex-compat
    %option noyywrap
    %option yylineno
    %{
    #include<stdio.h>
    #include<conio.h>
    #include<stdlib.h>
    #include<string.h>
    %}
    %{
    int INDENT=0,DEDENT=0,line=0;
    %}
    HASH    "#"
    NEWL    [\n]
    SPACE   " "
    MULTS   """
    COMP    "e^"
    LETTER  ([a-zA-Z])
    HEXL    ([a-fA-F1-9])
    ZERO    "0"
    EXP     "10^"
    COLON   ":"
    DOT     "."
    LPAREN  "("
    RPAREN  ")"
    PLUS    "+"
    MINUS   "-"
    SIGN    ({PLUS}|{MINUS})
    MULT    "*"
    DIV     "/"
    ASSIGN  "="
    EQUAL   "=="
    MORE_THAN ">"
    LESS_THAN "<"
    OR      "or"
    AND     "and"
    NOT     "not"
    IF      "if"
    ELSE    "else"
    LAMBDA  "lambda"
    FOR     "for"
    PASS    "pass"
    H       ({"h"|"H"})
    WHITESPACE  {[\t\n]}
    NUM_NZ      [1-9]
    NUM         [0-9]
    INTEGER     ({SIGN}?({ZERO}|({ZERO}+/({NUM_NZ}+))))
    REAL        ({INTEGER}{DOT}{NUM}|{INTEGER}{DOT}{NUM}{EXP}{INTEGER})
    HEX         ({PLUS}?(ZERO}|{HEXL}+{H}))
    COMPLEX     ({PLUS}?{REAL}+{COMP}{SIGN}?(({REAL}/"i")|"i"))
    ID          ("'"{.}+/{LETTER}+"'")
    STRING      ("'"{.}+"'")
    MULTISTRING ({MULTS}({.}+)|{NEWL}+{MULTS})
    COMMENT     ({.}+/{HASH})
    %%
   {SPACE}      {INDENT++; };
   {NEWL}   
    {
     if(INDENT>DEDENT) printf("Line %d: Found token %s (lexeme:       '%s').\n",yylineno,"INDENT",yytext);  
     if(INDENT<DEDENT)  printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"DEDENT",yytext);   
     DEDENT=INDENT; line++;};
    {INTEGER}       printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"INTEGER",yytext);
   {REAL}           printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"REAL",yytext);
    {HEX}           printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"HEX",yytext);
    {COMPLEX}       printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"COMPLEX",yytext);
    {STRING}        printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"STRING",yytext);
    {MULTISTRING}   printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"MULTISTRING",yytext);
    {IF}            printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"IF",yytext);
    {ELSE}          printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"ELSE",yytext);
    LAMBDA}     printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"LAMBDA",yytext);
    {FOR}           printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"FOR",yytext);
    {PASS}          printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"PASS",yytext);
    {ID}            printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"ID",yytext);
    {COLON}         printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"COLON",yytext);
    {LPAREN}        printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"LPAREN",yytext);
    {RAREN}         printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"RPAREN",yytext);
    {ARITH_OP}      printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"ARITH_OP",yytext);
    {REL_OP}        printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"REL_OP",yytext);
    {LOGIC_OP}      printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"LOGIC_OP",yytext);
    {NOT}           printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"NOT",yytext);
    {ASSIGN}        printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"ASSIGN",yytext);
    .       printf("Line %d: Invalid token  ('%s').\n",yylineno,yytext);
    %%
    int main(int argc,char* argv[])
    {
yylex();
    }

I have an unrecognized rule in line 102 which contains yylex();. What did I do wrong, and where is the error? I cannot find it.

hw1.text , line 102: unrecognized rule

when I try to compile it, I get this error. I'm writing flex hw1.text and the file and the flex are in the same folder.

Comments added based on deleted non-answer(s)

I did what you told me to do, for some reason I get an error for the 'printf' lines.

I've added :

ARITH_OP   ({PLUS}|{MINUS}|{DIV}|{MULT})
REL_OP     ({MORE_THAN}|{LESS_THAN}|{EQUAL})
LOGIC_OP   ({AND}|{OR})

What may be the problem?

Jonathan Leffler Jonathan Leffler · Accepted Answer · 2013-04-04T21:36:14

There are a number of problems that I see:

The action after {NEWL} should start on the same line as {NEWL}.

{NEWL}
{
if(INDENT>DEDENT) printf("Line %d: Found token %s (lexeme:       '%s').\n",yylineno,"INDENT",yytext);
if(INDENT<DEDENT)  printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"DEDENT",yytext);
DEDENT=INDENT; line++;};

This should be something like:

{NEWL} {
          if(INDENT>DEDENT) printf("Line %d: Found token %s (lexeme:       '%s').\n",yylineno,"INDENT",yytext);
          if(INDENT<DEDENT)  printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"DEDENT",yytext);
          DEDENT=INDENT; line++;
       };

Another problem is a missing { for the {LAMBDA} rule:

{ELSE}          printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"ELSE",yytext);
LAMBDA}     printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"LAMBDA",yytext);
{FOR}           printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"FOR",yytext);

You use {RAREN} instead of {RPAREN}.
You have not defined {ARITH_OP}, {REL_OP}, or {LOGIC_OP}.
I think the rule for MULTS should be something like this (but I might be wrong):
```
MULTS "\""
```

To get a clean compilation, I moved the following rules into a comment after the rules section of the scanner:

{MULTISTRING}   printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"MULTISTRING",yytext);
{ID}            printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"ID",yytext);
{STRING}        printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"STRING",yytext);
{HEX}           printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"HEX",yytext);
{COMPLEX}       printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"COMPLEX",yytext);
{INTEGER}       printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"INTEGER",yytext);
{REAL}          printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"REAL",yytext);
{ARITH_OP}      printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"ARITH_OP",yytext);
{REL_OP}        printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"REL_OP",yytext);
{LOGIC_OP}      printf("Line %d: Found token %s (lexeme: '%s').\n",yylineno,"LOGIC_OP",yytext);

You also, clearly, need a function in place of the repetitive printf() statements.

'Working' Code

This code compiles and runs. There are still lots of problems with the rules (for instance, the real numbers can only have a single digit after the decimal point, and identifiers must be enclosed in single quotes). It uses a print_token() function to make the scanner actions readable.

%option lex-compat
%option noyywrap
%option yylineno

%{
#include <stdio.h>
/*#include <conio.h>*/
#include <stdlib.h>
#include <string.h>
%}

%{
int INDENT=0, DEDENT=0, line=0;
static void print_token(const char *token);

/*
Original rules moved out of the way:

MULTISTRING ({MULTS}({.}+)|{NEWL}+{MULTS})
STRING      ("'"{.}+"'")
INTEGER     ({SIGN}?({ZERO}|({ZERO}+/({NUM_NZ}+))))
ID          ("'"{.}+/{LETTER}+"'")
H           ({"h"|"H"})
COMPLEX     ({PLUS}?{REAL}+{COMP}{SIGN}?(({REAL}/"i")|"i"))
MULTISTRING ({MULTS}({.}+)|{NEWL}+{MULTS})
*/

/*
Examples of patterns matched:

INTEGER: 1 +1 -2 0 99 999 987654321092344
REAL:    1.2 -2.1 3.14159
ID:      'ThisIsAnId'
HEX:     +0 DeadBeefH
*/
%}

HASH        "#"
NEWL        [\n]
SPACE       " "
MULTS       "\""
COMP        "e^"
LETTER      ([a-zA-Z])
HEXL        ([a-fA-F1-9])
ZERO        "0"
EXP         "10^"
COLON       ":"
DOT         "."
LPAREN      "("
RPAREN      ")"
PLUS        "+"
MINUS       "-"
SIGN        ({PLUS}|{MINUS})
MULT        "*"
DIV         "/"
ASSIGN      "="
EQUAL       "=="
MORE_THAN   ">"
LESS_THAN   "<"
OR          "or"
AND         "and"
NOT         "not"
IF          "if"
ELSE        "else"
LAMBDA      "lambda"
FOR         "for"
PASS        "pass"
H           ("h"|"H")
WHITESPACE  {[\t\n]}
NUM_NZ      [1-9]
NUM         [0-9]

INTEGER     ({SIGN}?{NUM_NZ}+)

REAL        ({INTEGER}{DOT}{NUM}|{INTEGER}{DOT}{NUM}{EXP}{INTEGER})
HEX         ({PLUS}?({ZERO}|{HEXL}+{H}))
COMPLEX     ({PLUS}?{REAL}+{COMP}{SIGN}?(({REAL}"i")|"i"))

ID          ("'"{LETTER}+"'")
STRING      ("'"[^']+"'")
MULTISTRING ({MULTS}([^"]+)|{NEWL}+{MULTS})
COMMENT     ({.}+/{HASH})
ARITH_OP    ({PLUS}|{MINUS}|{DIV}|{MULT})
REL_OP      ({MORE_THAN}|{LESS_THAN}|{EQUAL})
LOGIC_OP    ({AND}|{OR})

%%

{SPACE}         { INDENT++; };
{NEWL}          {
                    if (INDENT > DEDENT) print_token("INDENT");  
                    if (INDENT < DEDENT) print_token("DEDENT");   
                    DEDENT = INDENT;
                    line++;
                }
{MULTISTRING}   print_token("MULTISTRING");
{STRING}        print_token("STRING");
{HEX}           print_token("HEX");
{ID}            print_token("ID");
{INTEGER}       print_token("INTEGER");
{COMPLEX}       print_token("COMPLEX");
{REAL}          print_token("REAL");
{IF}            print_token("IF");
{ELSE}          print_token("ELSE");
{LAMBDA}        print_token("LAMBDA");
{FOR}           print_token("FOR");
{PASS}          print_token("PASS");
{COLON}         print_token("COLON");
{LPAREN}        print_token("LPAREN");
{RPAREN}        print_token("RPAREN");
{ARITH_OP}      print_token("ARITH_OP");
{REL_OP}        print_token("REL_OP");
{LOGIC_OP}      print_token("LOGIC_OP");
{NOT}           print_token("NOT");
{ASSIGN}        print_token("ASSIGN");
.               printf("Line %d: Invalid token ('%s').\n", yylineno, yytext);

%%

static void print_token(const char *token)
{
    printf("Line %d: Found token %s (lexeme: '%s').\n", yylineno, token, yytext);
}

int main(void)
{
    yylex();
}

One of the problems is the use of / in the rules. In Lex and Flex, IIRC, that introduces trailing context. It was not remotely clear to me what it was intended to do in the rules where it was used. There was at least one ZERO} where you needed {ZERO}. You seemed to be trying to use {.}+ as a synonym for [.]+ or maybe .+ (and I generally changed it to [^"]+ or [^']+ anyway.

The rule for COMMENT still contains a slash. It appears that a hash marks the end of a comment (but is not part of a comment), not the start of a comment; that is more than a little unusual (I think it is actually unusable) as a comment convention.

And as to how I made this work, I did it one bit at a time.

I moved the bits that caused compilation errors out of the way — into comments. When I'd got something that compiled, I tested it (on its own source code). Then I collected one of the broken rules, kept a record so I knew what the original was, and modified it so it made some sense to me. Rinse, repeat, one step at a time. I used a VCS (actually, git) as I was going, so I knew I could recover a 'working' version if I made a drastic mistake. You do use a VCS, don't you?

Unrecognized rule error

Comments added based on deleted non-answer(s)

1 Answers

'Working' Code