lexer

im using the python module ply.lex to write a lexer. I got some of my tokens specified with regular expression but now im stuck. I've a list of Keywords who should be a token . data is a list with about 1000 Keywords which should be all recognised as one sort of Keyword. This can be for example: _Function1 _UDFType2 and so on. All words in the list are separated by whitespaces thats it. I just want that lexer to recognise the words in this list, so that it would return a token of type `KEYWORD. data = 'Keyword1 Keyword2 Keyword3 Keyword4' def t_KEYWORD(t): # ... r'\$' + data ?? return t text =

i want to parse something like this in my lexer: ( begin expression ) where expressions are also surrounded by brackets. it isn't important what is in the expression, i just want to have all what's between the (begin and the matching ) as a token. an example would be: (begin (define x (+ 1 2))) so the text of the token should be (define x (+ 1 2))) something like PROGRAM : LPAREN BEGIN .* RPAREN; does (obviously) not work because as soon as he sees a ")", he thinks the rule is over, but i need the matching bracket for this. how can i do that? Bart Kiers Inside lexer rules, you can invoke rules

I'm trying to implement a tool for merging different versions of some source code. Given two versions of the same source code, the idea would be to parse them, generate the respective Abstract Source Trees (AST), and finally merge them into a single output source keeping grammatical consistency - the lexer and parser are those of question ANTLR: How to skip multiline comments . I know there is class ParserRuleReturnScope that helps... but getStop() and getStart() always return null :-( Here is a snippet that illustrates how I modified my perser to get rules printed: parser grammar

Background I want to be able to parse Javascript source in a Delphi Application. I need to be able to identify variables and functions within the source for the purpose of making changes to the code through later code. I understand that I probably need to use a lexer for this purpose but have not had much luck using the lexer which I found ( Dyaclexx ). Question Is there a suitable freeware or open source delphi parser/lexer which already has token sets for Javascript or could be easily modified for this purpose without too much trouble? If there isn't such a tool already available then what

If someone would clear my mind from the confusion behind look-ahead relation to tokenizing involving greery/non-greedy matching i'd be more than glad. Be ware this is a slightly long post because it's following my thought process behind. I'm trying to write antlr3 grammar that allows me to match input such as: "identifierkeyword" I came up with a grammar like so in Antlr 3.4: KEYWORD: 'keyword' ; IDENTIFIER : (options {greedy=false;}: (LOWCHAR|HIGHCHAR))+ ; /** lowercase letters */ fragment LOWCHAR : 'a'..'z'; /** uppercase letters */ fragment HIGHCHAR : 'A'..'Z'; parse: IDENTIFIER KEYWORD EOF

i am trying to learn more about compilers and programming languages, unfortunately my university doesnt offer a course about compilers and so i have to do myself (thank you internet). At the moment im tryin to understand and to implement a lexer for my language and i need regular expressions. I am used to script perl regex pretty quickly and i thought that i could embed Perl in my C++ lexer . Now the questions are: Will it cause Heavy overhead? Should i try to make peace with BOOST (or any other c++ library good gor regex) ? Thank you for reading this :) No reason you can't, part of being a