lexical-analysis

I'm trying to match a control character in the form \^c where c is any valid character for control characters. I have this regular expression, but it's not currently working: \\[^][@-z] I think the problem lies with the fact that the caret character (^) is part of the regular expressions parsing engine. Match an ASCII text string of the form ^X using the pattern \^. , nothing more. Match an ASCII text string of the form \^X with the pattern \\\^. . You may wish to constrain that dot to [?@_\[\]^\\] , so \\\^[A-Z?@_\[\]^\\] . It’s easier to read as [?\x40-\x5F] for the bracketed character class

Besides not closing a comment /*... , what constitutes a lexical error in C? Here are some: "abc<EOF> where EOF is the end of the file. In fact, EOF in the middle of many lexemes should produce errors: 0x<EOF> I assume that using bad escapes in strings is illegal: "ab\qcd" Probably trouble with floating point exponents 1e+% Arguably, you shouldn't have stuff at the end of a preprocessor directive: #if x % Basically anything that is not conforming to ISO C 9899/1999, Annex A.1 "Lexical Grammar" is a lexical fault if the compiler does its lexical analysis according to this grammar. Here are some

I'm looking for a decent lexical scanner generator for C#/.NET -- something that supports Unicode character categories, and generates somewhat readable & efficient code. Anyone know of one? EDIT: I need support for Unicode categories , not just Unicode characters. There are currently 1421 characters in just the Lu (Letter, Uppercase) category alone, and I need to match many different categories very specifically, and would rather not hand-write the character sets necessary for it. Also, actual code is a must -- this rules out things that generate a binary file that is then used with a driver

Hey folks, thanks for reading I am currently attempting to do a Google-style calculator. You input a string, it determines if it can be calculated and returns the result. I began slowly with the basics : + - / * and parenthesis handling. I am willing to improve the calculator over time, and having learned a bit about lexical analysis a while ago, I built a list of tokens and associated regular expression patterns. This kind of work is easily applicable with languages such as Lex and Yacc, except I am developping a Javascript-only application. I tried to transcript the idea into Javascript but

I am parsing sentences. I want to know the relevant content of each sentence, defined loosely as "semi-unique words" in relation to the rest of the corpus. Something similar to Amazon's "statistically improbable phrases", which seem to (often) convey the character of a book through oddball strings of words. My first pass was to start making a common words list. This knocks out the easy ones like a , the , from , etc. Obviously, it turns out that this list gets quite long. One idea is to generate this list: Make a histogram of the corpus' word frequencies, and lop off the top 10% or something

I realize this is a broad topic, but I'm looking for a good primer on parsing meaning from text, ideally in Python. As an example of what I'm looking to do, if a user makes a blog post like: "Manny Ramirez makes his return for the Dodgers today against the Houston Astros", what's a light-weight/ easy way of getting the nouns out of a sentence? To start, I think I'd limit it to proper nouns, but I wouldn't want to be limited to just that (and I don't want to rely on a simple regex that assumes anything Title Capped is a proper noun). To make this question even worse, what are the things I'm not