How to increase stack size in bison (and solve “memory exhausted”)

Question

My bison-based parser started choking on some moderately sized files I've generated recently.

It throws an exception about "memory exhausted."

The bison man page says that this is likely due to use of right-hand recursion. Without trying to rewrite the grammar (I am on a tight deadline), I would like to simply increase the stack to get the parser to parse this file. I tried to follow the bison man page and #define YYMAXDEPTH to some number larger than the default 10000, but that didn't work. When I look at the output from bison, it seems like YYMAXDEPTH is only conditionally used when YYSTACK_RELOCATE is defined and YYSTACK_RELOCATE is defined only in this case:

#if (! defined yyoverflow \
     && (! defined __cplusplus \
         || (defined YYLTYPE_IS_TRIVIAL && YYLTYPE_IS_TRIVIAL \
             && defined YYSTYPE_IS_TRIVIAL && YYSTYPE_IS_TRIVIAL)))

Since I am compiling with C++, the above #ifdef fails and thus YYSTACK_RELOCATE is not defined.

Is this a bug or a feature? Anybody knows what's the right way to increase the stack size?

BTW, this is the snippet of the bison-generated code where the stack overflows:

....
....
if (yyss + yystacksize - 1 <= yyssp)
{
  /* Get the current used size of the three stacks, in elements.  */
  YYSIZE_T yysize = yyssp - yyss + 1;

#ifdef yyoverflow
  {
    /* Give user a chance to reallocate the stack.  Use copies of
       these so that the &'s don't force the real ones into
       memory.  */
    YYSTYPE *yyvs1 = yyvs;
    yytype_int16 *yyss1 = yyss;
    YYLTYPE *yyls1 = yyls;

    /* Each stack pointer address is followed by the size of the
       data in use in that stack, in bytes.  This used to be a
       conditional around just the two extra args, but that might
       be undefined if yyoverflow is a macro.  */
    yyoverflow (YY_("memory exhausted"),
                &yyss1, yysize * sizeof (*yyssp),
                &yyvs1, yysize * sizeof (*yyvsp),
                &yyls1, yysize * sizeof (*yylsp),
                &yystacksize);

    yyls = yyls1;
    yyss = yyss1;
    yyvs = yyvs1;
  }
#else /* no yyoverflow */
# ifndef YYSTACK_RELOCATE
  goto yyexhaustedlab;             // <====== Overflows and throws exception here
# else
  ......
  ......

Answer 1

In general, C++ objects -- unlike C datatypes -- cannot be relocated by memcpy. Consequently, bison refuses to relocate its stack unless it somehow knows that the stack objects are "trivial", and if the objects are C++ objects, it assumes that they are not.

YYMAXDEPTH is the maximum stack that the parser will allocate, but the initial stack size is YYINITDEPTH. Since C++ stacks cannot be relocated, the initial size must be large enough for any input, so you need to increase YYINITDEPTH, not YYMAXDEPTH.

Alternatively, you could figure out how to tell bison that the C++ stack objects are relocatable (if they are), or you could try a more recent version of bison: allegedly, v3 is more willing to let you burn yourself, but I haven't tried it myself. Finally, you can define yyoverflow to provide your own stack relocation mechanism: unfortunately, this is undocumented and unnecessarily complicated.