How to NER and POS tag a pre-tokenized text with Stanford CoreNLP?

Question

I'm using the Stanford's CoreNLP Named Entity Recognizer (NER) and Part-of-Speech (POS) tagger in my application. The problem is that my code tokenizes the text beforehand and then I need to NER and POS tag each token. However I was only able to find out how to do that using the command line options but not programmatically.

Can someone please tell me how programmatically can I NER and POS tag pretokenized text using Stanford's CoreNLP?

Edit:

I'm actually using the individual NER and POS instructions. So my code was written as instructed in the tutorials given in the Stanford's NER and POS packages. But I have CoreNLP in my classpath. So I have the CoreNLP in my classpath but using the tutorials in the NER and POS packages.

Edit:

I just found that there are instructions as how one can set the properties for CoreNLP here http://nlp.stanford.edu/software/corenlp.shtml but I wish if there was a quick way to do what I want with Stanford NER and POS taggers so I don't have to recode everything!

Answer 1

If you set the property:

tokenize.whitespace = true

then the CoreNLP pipeline will tokenize on whitespace rather than the default PTB tokenization. You may also want to set:

ssplit.eolonly = true

so that you only split sentences on newline characters.

Answer 2

To programmatically run a classifier over a list of tokens that you've already gotten via some other means, without a kludge like pasting them together with whitespace and then tokenizing again, you can use the Sentence.toCoreLabelList method:

String[] token_strs = {"John", "met", "Amy", "in", "Los", "Angeles"};
List<CoreLabel> tokens = edu.stanford.nlp.ling.Sentence.toCoreLabelList(token_strs);
for (CoreLabel cl : classifier.classifySentence(tokens)) {
  System.out.println(cl.toShorterString());
}

Output:

[Value=John Text=John Position=0 Answer=PERSON Shape=Xxxx DistSim=463]
[Value=met Text=met Position=1 Answer=O Shape=xxxk DistSim=476]
[Value=Amy Text=Amy Position=2 Answer=PERSON Shape=Xxx DistSim=396]
[Value=in Text=in Position=3 Answer=O Shape=xxk DistSim=510]
[Value=Los Text=Los Position=4 Answer=LOCATION Shape=Xxx DistSim=449]
[Value=Angeles Text=Angeles Position=5 Answer=LOCATION Shape=Xxxxx DistSim=199]