stanford-nlp

I'm using the Stanford Named Entity Recognizer http://nlp.stanford.edu/software/CRF-NER.shtml and it's working fine. This is List<List<CoreLabel>> out = classifier.classify(text); for (List<CoreLabel> sentence : out) { for (CoreLabel word : sentence) { if (!StringUtils.equals(word.get(AnswerAnnotation.class), "O")) { namedEntities.add(word.word().trim()); } } } However the problem I'm finding is identifying names and surnames. If the recognizer encounters "Joe Smith", it is returning "Joe" and "Smith" separately. I'd really like it to return "Joe Smith" as one term. Could this be achieved

I want to compute how similar two arbitrary sentences are to each other. For example: A mathematician found a solution to the problem. The problem was solved by a young mathematician. I can use a tagger, a stemmer, and a parser, but I don’t know how detect that these sentences are similar. These two sentences are not just similar, they are almost paraphrases , i.e., two alternative ways of expressing the same meaning. It is also a very simple case of paraphrase, in which both utterances use the same words with the only exception of one being in active form while the other is passive. (The two

I'm using some NLP libraries now, (stanford and nltk) Stanford I saw the demo part but just want to ask if it possible to use it to identify more entity types. So currently stanford NER system (as the demo shows) can recognize entities as person(name), organization or location. But the organizations recognized are limited to universities or some, big organizations. I'm wondering if I can use its API to write program for more entity types, like if my input is "Apple" or "Square" it can recognize it as a company. Do I have to make my own training dataset? Further more, if I ever want to extract

Recently I have been trying to train n-gram entities with Stanford Core NLP. I have followed the following tutorials - http://nlp.stanford.edu/software/crf-faq.shtml#b With this, I am able to specify only unigram tokens and the class it belongs to. Can any one guide me through so that I can extend it to n-grams. I am trying to extract known entities like movie names from chat data set. Please guide me through in case I have mis-interpretted the Stanford Tutorials and the same can be used for the n-gram training. What I am stuck with is the following property #structure of your training file;

I'm trying to use the Stanford Segementer bit from the NLTK Tokenize package. However, I run into issues just trying to use the basic test set. Running the following: # -*- coding: utf-8 -*- from nltk.tokenize.stanford_segmenter import StanfordSegmenter seg = StanfordSegmenter() seg.default_config('zh') sent = u'这是斯坦福中文分词器测试' print(seg.segment(sent)) Results in this error: I got as far as to add... import os javapath = "C:/Users/User/Folder/stanford-segmenter-2017-06-09/*" os.environ['CLASSPATH'] = javapath ...to the front of my code, but that didn't seem to help. How do I get the segmentor to

import nltk from itertools import groupby def get_continuous_chunks(tagged_sent): continuous_chunk = [] current_chunk = [] for token, tag in tagged_sent: if tag != "O": current_chunk.append((token, tag)) else: if current_chunk: # if the current chunk is not empty continuous_chunk.append(current_chunk) current_chunk = [] # Flush the final current_chunk into the continuous_chunk, if any. if current_chunk: continuous_chunk.append(current_chunk) return continuous_chunk ne_tagged_sent = [('Rami', 'PERSON'), ('Eid', 'PERSON'), ('is', 'O'), ('studying', 'O'), ('at', 'O'), ('Stony', 'ORGANIZATION'), (