lexical-analysis

I'm trying to find out if there is a known algorithm that can detect the "key concept" of a sentence. The use case is as follows: User enters a sentence as a query (Does chicken taste like turkey?) Our system identifies the concepts of the sentence (chicken, turkey) And it runs a search of our corpus content The area that we're lacking in is identifying what the core "topic" of the sentence is really about. The sentence "Does chicken taste like turkey" has a primary topic of "chicken", because the user is asking about the taste of chicken. While "turkey" is a helper topic of less importance.

I'm working on a project where I need to analyze a page of text and collections of pages of text to determine dominant words. I'd like to know if there is a library (prefer c# or java) that will handle the heavy lifting for me. If not, is there an algorithm or multiple that would achieve my goals below. What I want to do is similar to word clouds built from a url or rss feed that you find on the web, except I don't want the visualization. They are used all the time for analyzing the presidential candidate speeches to see what the theme or most used words are. The complication, is that I need

I'm trying to write a lexer in JavaScript for finding tokens of a simple domain-specific language. I started with a simple implementation which just tries to match subsequent regexps from the current position in a line to find out whether it matches some token format and accept it then. The problem is that when something doesn't match inside such regexp, the whole regexp fails, so I don't know which character exactly caused it to fail. Is there any way to find out the position in the string which caused the regular expression to fail? INB4: I'm not asking about debugging my regexp and

I know question Lex and Yacc in PHP was asked before but 1 year ago. Is there any new mature PHP parser generator now? My searches drove me to the following ones, what do you think about them, any others? code.google.com/p/antlrphpruntime/ : The ANTLR PHP version but it seems to be very beta version and I think there is a lot of work to do. The advantage is that I can write the grammar in the ANTLR Works tool. pear.php.net/package/PHP_ParserGenerator/docs/0.1.7/ : I tried but it seems very complicated, to be used with the PHP_LexerGenerator. sourceforge.net/projects/lime-php/ : I didn't try