add stemming support to CountVectorizer (sklearn)

Question

I\'m trying to add stemming to my pipeline in NLP with sklearn.

from nltk.stem.snowball import FrenchStemmer

stop = stopwords.words(\'french\')
stemmer = French         


        

Answer 1

You can try:

def build_analyzer(self):
    analyzer = super(CountVectorizer, self).build_analyzer()
    return lambda doc:(stemmer.stem(w) for w in analyzer(doc))

and remove the __init__ method.