stemming

I have a file 'check_text.txt' that contains " said say says make made ". I'd like to perform stemming on it to get "say say say make make". I tried to use stemDocument in tm package, as the following, but only get "said say say make made". Is there a way to perform stemming on past tense words? Is it necessary to do so in real-world natural language processing? Thanks! filename = 'check_text.txt' con <- file(filename, "rb") text_data <- readLines(con,skipNul = TRUE) close(con) text_VS <- VectorSource(text_data) text_corpus <- VCorpus(text_VS) text_corpus <- tm_map(text_corpus, stemDocument,

I am building a little search function for my site. I am taking my user's query, stemming the keywords and then running a fulltext MySQL search against the stemmed keywords. The problem is that MySQL is treating the stems as literal. Here is the process that is happening: user searches for a word like "baseballs" my stemming algorithm (Porter Stemmer) turns "baseballs" into "basebal" fulltext does not find anything matching "basebal", even though there SHOULD be matches for "baseball" and "baseballs" How do I do the equivalent of LIKE 'basebal%' with fulltext? EDIT: Here is my current query:

Given some (English) word that we shall assume is a plural , is it possible to derive the singular form? I'd like to avoid lookup/dictionary tables if possible. Some examples: Examples -> Example a simple 's' suffix Glitch -> Glitches 'es' suffix, as opposed to above Countries -> Country 'ies' suffix. Sheep -> Sheep no change: possible fallback for indeterminate values Or, this seems to be a fairly exhaustive list. Suggestions of libraries in language x are fine, as long as they are open-source (ie, so that someone can examine them to determine how to do it in language y ) It really depends on

I am looking for a class or method that takes a long string of many 100s of words and tokenizes, removes the stop words and stems for use in an IR system. For example: "The big fat cat, said 'your funniest guy i know' to the kangaroo..." the tokenizer would remove the punctuation and return an ArrayList of words the stop word remover would remove words like "the", "to", etc the stemmer would reduce each word the their 'root', for example 'funniest' would become funny Many thanks in advance. jitter AFAIK Lucene can do what you want. With StandardAnalyzer and StopAnalyzer you can to the stop

I am doing a lot of analysis with the TM package. One of my biggest problems are related to stemming and stemming-like transformations. Let's say I have several accounting related terms (I am aware of the spelling issues). After stemming we have: accounts -> account account -> account accounting -> account acounting -> acount acount -> acount acounts -> acount accounnt -> accounnt Result : 3 Terms (account, acount, account) where I would have liked 1 (account) as all these relate to the same term. 1) To correct spelling is a possibility, but I have never attempted that in R. Is that even

I'm looking for a Java library to extract keywords from a block of text. The process should be as follows: stop word cleaning -> stemming -> searching for keywords based on English linguistics statistical information - meaning if a word appears more times in the text than in the English language in terms of probability than it's a keyword candidate. Is there a library that performs this task? sp00m Here is a possible solution using Apache Lucene . I didn't use the last version but the 3.6.2 one , since this is the one I know the best. Besides the /lucene-core-x.x.x.jar , don't forget to add