spacy

问题 I want to merge some rows if they are nearly similar. Similarity can be checked by using spaCy. df: string yellow color yellow color looks like yellow color bright red color okay red color blood output: string yellow color looks like bright red color okay blood solution: brute force approach is - for every item in string check similarity with other n-1 item if greater than some threshold value then merge. Is there any other approach ? As i am not in contact with much people idk how they do it

问题 I want to merge some rows if they are nearly similar. Similarity can be checked by using spaCy. df: string yellow color yellow color looks like yellow color bright red color okay red color blood output: string yellow color looks like bright red color okay blood solution: brute force approach is - for every item in string check similarity with other n-1 item if greater than some threshold value then merge. Is there any other approach ? As i am not in contact with much people idk how they do it

问题 When using spacy you can easily loop across the noun_phrases of a text as follows: S='This is an example sentence that should include several parts and also make clear that studying Natural language Processing is not difficult' nlp = spacy.load('en_core_web_sm') doc = nlp(S) [chunk.text for chunk in doc.noun_chunks] # = ['an example sentence', 'several parts', 'Natural language Processing'] You can also get the "root" of the noun chunk: [chunk.root.text for chunk in doc.noun_chunks] # = [

问题 Using Python Spacy, I am trying to extract entities from multiple subject passive voice sentence. Sentence = "John and Jenny were accused of crimes by David" My intention is to extract both "John and Jenny” from the sentence as nsubjpass and .ent_ . However, I am only able to extract “John” as nsubjpass. How to extract both them? Notice that while John is found as an entity in .ents, Jenny is considered as conj instead of nsubjpass. How to improve it? code each_sentence3 = "John and Jenny

问题 In the spacy's text classification train_textcat example, there are two labels specified Positive and Negative . Hence the cats score is represented as cats = [{"POSITIVE": bool(y), "NEGATIVE": not bool(y)} for y in labels] I am working with Multilabel classfication which means i have more than two labels to tag in one text. I have added my labels as textcat.add_label("CONSTRUCTION") and to specify cats score I have used cats = [{"POSITIVE": bool(y), "NEGATIVE": not bool(y)} for y in labels]

问题 Edit due to off-topic I want to use regex in SpaCy to find any combination of (Accrued or accrued or Annual or annual) leave by this code: from spacy.matcher import Matcher nlp = spacy.load('en_core_web_sm') matcher = Matcher(nlp.vocab) # Add the pattern to the matcher matcher.add('LEAVE', None, [{'TEXT': {"REGEX": "(Accrued|accrued|Annual|annual)"}}, {'LOWER': 'leave'}]) # Call the matcher on the doc doc= nlp('Annual leave shall be paid at the time . An employee is to receive their annual

问题 Edit due to off-topic I want to use regex in SpaCy to find any combination of (Accrued or accrued or Annual or annual) leave by this code: from spacy.matcher import Matcher nlp = spacy.load('en_core_web_sm') matcher = Matcher(nlp.vocab) # Add the pattern to the matcher matcher.add('LEAVE', None, [{'TEXT': {"REGEX": "(Accrued|accrued|Annual|annual)"}}, {'LOWER': 'leave'}]) # Call the matcher on the doc doc= nlp('Annual leave shall be paid at the time . An employee is to receive their annual

问题 I used this official example code to train a NER model from scratch using my own training samples. When I predict using this model on new text, I want to get the probability of prediction of each entity. # test the saved model print("Loading from", output_dir) nlp2 = spacy.load(output_dir) for text, _ in TRAIN_DATA: doc = nlp2(text) print("Entities", [(ent.text, ent.label_) for ent in doc.ents]) print("Tokens", [(t.text, t.ent_type_, t.ent_iob) for t in doc]) I am unable to find a method in

问题 lemmatization on pronouns via [token.lemma_ for token in doc] gives lemmatized word for pronouns as -PRON- , is this a bug? 回答1: No, this is in fact intended behaviour. See the documentation here: Unlike verbs and common nouns, there's no clear base form of a personal pronoun. Should the lemma of "me" be "I", or should we normalize person as well, giving "it" — or maybe "he"? spaCy's solution is to introduce a novel symbol, -PRON- , which is used as the lemma for all personal pronouns. It

问题 I am extending a spaCy model using rules. While looking through the documentation, I noticed the IN attribute, which is used to map patterns to a dictionary of properties. This is great however it only works on single tokens. For example, this pattern: {"label":"EXAMPLE","pattern":[{"LOWER": {"IN": ["such as", "like", "for example"]}}]} will only work with the term like but not the others. What is the best way to achieve the same result for multi-terms attributes? 回答1: It depends on how