n-gram

I have a list of documents that have already been tokenized: dat <- list(c("texaco", "canada", "lowered", "contract", "price", "pay", "crude", "oil", "canadian", "cts", "barrel", "effective", "decrease", "brings", "companys", "posted", "price", "benchmark", "grade", "edmonton", "swann", "hills", "light", "sweet", "canadian", "dlrs", "bbl", "texaco", "canada", "changed", "crude", "oil", "postings", "feb", "reuter"), c("argentine", "crude", "oil", "production", "pct", "january", "mln", "barrels", "mln", "barrels", "january", "yacimientos", "petroliferos", "fiscales", "january", "natural", "gas",

I found this solution on SO to detect n-grams in a string: (here: N-gram generation from a sentence ) import java.util.*; public class Test { public static List<String> ngrams(int n, String str) { List<String> ngrams = new ArrayList<String>(); String[] words = str.split(" "); for (int i = 0; i < words.length - n + 1; i++) ngrams.add(concat(words, i, i+n)); return ngrams; } public static String concat(String[] words, int start, int end) { StringBuilder sb = new StringBuilder(); for (int i = start; i < end; i++) sb.append((i > start ? " " : "") + words[i]); return sb.toString(); } public static

At least 3 types of n-grams can be considered for representing text documents: byte-level n-grams character-level n-grams word-level n-grams It's unclear to me which one should be used for a given task (clustering, classification, etc). I read somewhere that character-level n-grams are preferred to word-level n-grams when the text contains typos, so that "Mary loves dogs" remains similar to "Mary lpves dogs". Are there other criteria to consider for choosing the "right" representation? Evaluate . The criterion for choosing the representation is whatever works . Indeed, character level (!=

I am trying tokenize strings into ngrams. Strangely in the documentation for the NGramTokenizer I do not see a method that will return the individual ngrams that were tokenized. In fact I only see two methods in the NGramTokenizer class that return String Objects. Here is the code that I have: Reader reader = new StringReader("This is a test string"); NGramTokenizer gramTokenizer = new NGramTokenizer(reader, 1, 3); Where are the ngrams that were tokenized? How can I get the output in Strings/Words? I want my output to be like: This, is, a, test, string, This is, is a, a test, test string, This