nlp

问题 I am cleaning a column in my data frame , Sumcription, and am trying to do 3 things: Tokenize Lemmantize Remove stop words import spacy nlp = spacy.load('en_core_web_sm', parser=False, entity=False) df['Tokens'] = df.Sumcription.apply(lambda x: nlp(x)) spacy_stopwords = spacy.lang.en.stop_words.STOP_WORDS spacy_stopwords.add('attach') df['Lema_Token'] = df.Tokens.apply(lambda x: " ".join([token.lemma_ for token in x if token not in spacy_stopwords])) However, when I print for example: df.Lema

问题 For those who are not familiar with what a homophone is, I provide the following examples: our & are hi & high to & too & two While using the Speech API included with iOS, I am encountering situations where a user may say one of these words, but it will not always return the word I want. I looked into the [alternativeSubstrings] (link) property wondering if this would help, but in my testing of the above words, it always comes back empty. I also looked into the Natural Language API, but could

问题 How to do preprocessing steps like Stopword removal , punctuation removal , stemming and lemmatization in spaCy using python. I have text data in csv file like paragraphs and sentences. I want to do text cleaning. Kindly give example by loading csv in pandas dataframe 回答1: This may helps who is looking for answer for this quesion. import spacy #load spacy nlp = spacy.load("en", disable=['parser', 'tagger', 'ner']) stops = stopwords.words("english") def normalize(comment, lowercase, remove

问题 How to do preprocessing steps like Stopword removal , punctuation removal , stemming and lemmatization in spaCy using python. I have text data in csv file like paragraphs and sentences. I want to do text cleaning. Kindly give example by loading csv in pandas dataframe 回答1: This may helps who is looking for answer for this quesion. import spacy #load spacy nlp = spacy.load("en", disable=['parser', 'tagger', 'ner']) stops = stopwords.words("english") def normalize(comment, lowercase, remove

问题 How to do preprocessing steps like Stopword removal , punctuation removal , stemming and lemmatization in spaCy using python. I have text data in csv file like paragraphs and sentences. I want to do text cleaning. Kindly give example by loading csv in pandas dataframe 回答1: This may helps who is looking for answer for this quesion. import spacy #load spacy nlp = spacy.load("en", disable=['parser', 'tagger', 'ner']) stops = stopwords.words("english") def normalize(comment, lowercase, remove

问题 I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing . It is mentioned that it covers a wider spectrum of Out-Of-Vocabulary (OOV) words. Can someone please help me explain how WordPiece tokenization is actually done, and how it handles effectively helps to rare/OOV words? 回答1: WordPiece and BPE are two similar and commonly used techniques to segment words into subword-level in NLP tasks. In both cases,

问题 I would like to use Space to extract word relation information in the form of "agent, action, and patient." For example, "Autonomous cars shift insurance liability toward manufacturers" -> ("autonomous cars", "shift", "liability") or ("autonomous cars", "shift", "liability towards manufacturers"). In other words, "who did what to whom" and "what applied the action to something else." I don't know much about my input data, so I can't make many assumptions. I also want to extract logical

问题 I am trying to analyze a series of sentences by identifying the most common adverb-adjective-noun strings. I have managed to get answers for how to do so with random words but I think this is a standalone question, and it might better to be dealt with separately. In this case, I would like to omit common word types like personal pronouns, articles, prepositions and even verbs. Ideally, the results should produce: Most common nouns Most common adjectives Most common adverbs Most common

问题 i'm working on an Multi-Label Emotion Classification problem to be solved by word2vec. this is my code that i've learned from a couple of tutorials. now the accuracy is very low. about 0.02 which is telling me something is wrong in my code. but i cannot find it. i tried this code for TF-IDF and BOW (obviously except word2vec part) and i got much better accuracy scores such as 0.28, but it seems this one is somehow wrong: np.set_printoptions(threshold=sys.maxsize) wv = gensim.models

问题 I want to exclude 'The', 'They' and 'My' from being displayed in my word cloud. I'm using the python library 'wordcloud' as below, and updating the STOPWORDS list with these 3 additional stopwords, but the wordcloud is still including them. What do I need to change so that these 3 words are excluded? The libraries I imported are: import numpy as np import pandas as pd from wordcloud import WordCloud, STOPWORDS import matplotlib.pyplot as plt I've tried adding elements to the STOPWORDS set at