Switch input stream in lex and yacc

1
votes

I insist on using lex and not the flex.

am developing an API in lex like the one existing in the flex util ( yy_switch_buffer, yy_create_buffer ...) offering the possibility to switch between multiple buffers .

This is the main difficulty for me until now :

  • for example when I encounter a #include token i should switch the buffer to the included file. So first i should interrupt the current parsing action( I tried fclose(yyin) FAILED) the parser complete the whole current yyin. NOT good because I should parse the included file to store structures (for example ) used in the main file.

Question is : How can I interrupt immediately a parser ? Is it enough for me to define a new buffer using yyin = fopen(somefile, "r"); ??

4
yacclex
Why do you need to go through the pain of reimplementing what is already implemented elsewhere? - Jonathan Leffler
@JonathanLeffler i don't have the choice, am developing my solution for multiple platform(solaris, aix, win) and for some of them we can't install flex because the native lex existing solution may encounter problems. - Aymanadou
That makes little sense. The native lex may have problems, so you must use it? You could compile with flex on one machine and simply compile the Flex library (-lfl) on the native machine, plus the transferred C source generated from the Flex. But flex can be transported to those machines trivially. There is no 'native Lex' on Windows. You can either use MKS Lex & Yacc or Cygwin with Flex (and Bison if you need grammars too). - Jonathan Leffler

4 Answers

1
votes

It is going to be tough to handle it, if it is doable at all. AFAIK, Lex only allows you to switch input streams on EOF (real or simulated) when it calls yywrap().

Maybe you can fake things so that when you find the 'include' directive, you fake an EOF on the current stream and then have yywrap() fix things so that the new input comes from the included file, and then when you reach EOF on the included file, you have yywrap() restore input from the original input stream at the original position. Clearly, this works for nested includes (if it works at all) unless you arbitrarily restrict the number of levels of inclusion.

1
votes

There's no portable way to do this with POSIX lex -- different implementations have different internal arrangements of how they deal with and buffer input, and during lexing, may have read ahead of the currently processing token and buffered a bunch of the input. So you need to get it to save what it has currently buffered and switch to a different input, and then restore the buffered stuff (so it will be read next) after you're done with the #include or whatever. This is precisely what flex's buffer management calls are for, but if you insist on using lex, you'll need to (effectively) port these routines to understand the internals of whatever versions of lex you need to support.

1
votes

The solution to the "included input files" is part of flex documentation, which provides an example how to switch between flex inputs ftp://ftp.gnu.org/old-gnu/Manuals/flex-2.5.4/html_mono/flex.html#SEC12 ("Multiple Input Buffers")

0
votes

You can find flex tool ported to windows system here http://sourceforge.net/projects/winflexbison/