boost-spirit-lex

I'm writing a DSL and using a Boost Spirit lexer to tokenize my input. In my grammar, I want a rule similar to this (where tok is the lexer): header_block = tok.name >> ':' >> tok.stringval > ';' >> tok.description >> ':' >> tok.stringval > ';' ; Rather than specifying reserved words for the language (e.g. "name", "description") and deal with synchronizing these between the lexer and grammar, I want to just tokenize everything that matches [a-zA-Z_]\w* as a single token type (e.g. tok.symbol ), and let the grammar sort it out. If I weren't using a lexer, I might do something like this:

I'm wondering if there's a way in boost::spirit::lex to write a token value back to the input stream (possibly after editing) and rescanning again. What I'm basically looking for is a functionality like that offered by unput() in Flex. Thanks! Sounds like you just want to accept tokens in different orders but with the same meaning. Without further ado, here is a complete sample that shows how this would be done, exposing the identifier regardless of input order. Output: Input 'abc(' Parsed as: '(abc' Input '(abc' Parsed as: '(abc' Code #include <boost/spirit/include/qi.hpp> #include <boost

I want to tokenize my own extension of SQL syntax. This involves recognizing an escaped double quote inside a double quoted string. E.g. in MySQL these two string tokens are equivalent: """" (the second double quote acts as an escape character) and '"' . I have tried different things but I am stuck at how to replace a token's value. #include <boost/spirit/include/lex_lexertl.hpp> namespace lex = boost::spirit::lex; template <typename Lexer> struct sql_tokens : lex::lexer<Lexer> { sql_tokens() { string_quote_double = "\\\""; // '"' this->self("INITIAL") = string_quote_double [ lex::_state =

I try to implement a Lexer for a little programming language with Boost Spirit. I have to get the value of a token and I get a bad_get exception : terminate called after throwing an instance of 'boost::bad_get' what(): boost::bad_get: failed value get using boost::get Aborted I obtain this exception when doing : std::string contents = "void"; base_iterator_type first = contents.begin(); base_iterator_type last = contents.end(); SimpleLexer<lexer_type> lexer; iter = lexer.begin(first, last); end = lexer.end(); std::cout << "Value = " << boost::get<std::string>(iter->value()) << std::endl; My

I'm writing a simple expressions parser. It is build on a Boost.Spirit.Qi grammar based on Boost.Spirit.Lex tokens (Boost in version 1.56). The tokens are defined as follows: using namespace boost::spirit; template< typename lexer_t > struct tokens : lex::lexer<lexer_t> { tokens() : /* ... */, variable("%(\\w+)") { this->self = /* ... */ | variable; } /* ... */ lex::token_def<std::string> variable; }; Now I would like the variable token value to be just the name (the matching group (\\w+) ) without prefix % symbol. How do I do that? Using a matching group by itself doesn't help. Still value is

This does not compile (code below). There was another question here with the same error. But I don't understand the answer. I already tried inserting qi::eps in places -- but without success. I also tried already adding meta functions (boost::spirit::raits::is_container) for the types used -- but this also does not help. I also tried using the same variant containing all types I need to use everywhere. Same problem. Has anybody gotten this working for a lexer returning something else than double or int or string? And for the parser also returning non-trivial objects? I've tried implementing