Why are multi-line comments in flex/bison so evasive?

Question

I'm trying to parse C-style multi-line comments in my flex (.l) file:

%s ML_COMMENT
%%

...

<INITIAL>"/*"                   BEGIN(ML_COMMENT);
<ML_COMMENT>"*/"                BEGIN(INITIAL);  
<ML_COMMENT>[.\n]+              { }

I'm not returning any token and my grammar (.y) doesn't address comments in any way.

When I run my executable, I get a parse error:

$ ./a.out
/*
abc 
def
Parse error: parse error
$ echo "/* foo */" | ./a.out
Parse error: parse error

(My yyerror function does a printf("Parse error: %s\n"), which is where the first half of the redundant error message comes from).

I can see why the second example fails since the entirety of the input is a comment, and since comments are ignored by the grammar, there are no statements. Thus the input isn't a valid program. But the first part throws a parse error before I even finish the comment.

Also confusing:

$ ./a.out
/* foo */
a = b;
Parse error: parse error

In this case, the comment is closed prior to actual valid input (which, without the comment, parses just fine). The failure actually occurs after parsing "a", not after attempting to parse the assignment "a = b;". If I enter "a" on its own line, it still throws an error.

Given that the error message is a parser error and not a scanner error, is there something crucial I'm missing in my .y file? Or am I doing something wrong in my scanner rules that propagates over to the parser side?

EDIT: Per @Rudi's suggestion, I turned on debugging and found:

$ ./a.out
Starting parse
Entering state 0
Reading a token: /*
foo
Next token is 44 (IDENTIFER)
Shifting token 44 (IDENTIFER), Entering state 4
Reducing via rule 5 (line 130), IDENTIFER  -> identifier
state stack now 0
Entering state 5

I turned off debugging and found that /* foo */ = bar; indeed parses the same as foo = bar;. I'm using flex 2.5.4; it doesn't give me any warnings about the stateful rules I'm attempting to use.

Answer 1

I think you need to declare your ML_COMMENT start condition as an exclusive start condition so only the ML_COMMENT rules are active. %x ML_COMMENT instead of %s ML_COMMENT

Otherwise rules with no start conditions are also active.

Answer 2

Parsing comments this way can lead to errors because:

• you need to add conditions to all of your lex rules
• it becomes even more complex if you also want to handle // comments
• you still have the risk that yacc/bison merges two comments including everything in between

In my parser, I handle comments like this. First define lex rules for the start of the comment, like this:

\/\*     {
         if (!SkipComment())
            return(-1);
         }

\/\/     {
         if (!SkipLine())
            return(-1);
         }

then write the SkipComment and SkipLine functions. They need to consume all the input until the end of the comment is found (this is rather old code so forgive me the somewhat archaic constructions):

bool SkipComment (void)
{
int Key;

Key=!EOF;
while (true)
   {
   if (Key==EOF)
      {
      /* yyerror("Unexpected EOF within comment."); */
      break;
      }
   switch ((char)Key)
      {
      case '*' :
         Key=input();
         if (char)Key=='/') return true;
         else               continue;
         break;
      case '\n' :
         ++LineNr;
         break;
      }
   Key=input();
   }

return false;
}

bool SkipLine (void)
{
int Key;

Key=!EOF;
while (true)
   {
   if (Key==EOF)
      return true;
   switch ((char)Key)
      {
      case '\n' :
         unput('\n');
         return true;
         break;
      }
   Key=input();
   }

return false;
}

Answer 3

Besides the problem with %x vs %s, you also have the problem that the . in [.\n] matches (only) a literal . and not 'any character other than newline' like a bare . does. You want a rule like

<ML_COMMENT>.|"\n"     { /* do nothing */ }

instead

Answer 4

I found this description of the C language grammar (actually just the lexer) very useful. I think it is mostly the same as Patrick's answer, but slightly different.

http://www.lysator.liu.se/c/ANSI-C-grammar-l.html