Is it advisable to use tokens for the purpose of syntax highlighting?
I'm trying to implement syntax highlighting in C# on Android, using Xamarin. I'm using the ANTLR v4 library for C# to achieve this. My code, which is currently syntax highlighting Java with this grammar , does not attempt to build a parse tree and use the visitor pattern. Instead, I simply convert the input into a list of tokens: private static IList<IToken> Tokenize(string text) { var inputStream = new AntlrInputStream(text); var lexer = new JavaLexer(inputStream); var tokenStream = new CommonTokenStream(lexer); tokenStream.Fill(); return tokenStream.GetTokens(); } Then I loop through all of