sklearn TfidfVectorizer : Generate Custom NGrams by not removing stopword in them

Question

Following is my code:

sklearn_tfidf = TfidfVectorizer(ngram_range= (3,3),stop_words=stopwordslist, norm=\'l2\',min_df=0, use_idf=True, smooth_idf=False, subl         


        

Answer 1

Yes, you need to supply your own analyzer function which will convert the documents to the features as per your requirements.

According to the documentation:

analyzer : string, {‘word’, ‘char’, ‘char_wb’} or callable

....
....
If a callable is passed it is used to extract the sequence of 
features out of the raw, unprocessed input.

In that custom callable you need to take care of first splitting the sentence into different parts, removing special chars like comma, braces, symbols etc, convert them to lower case, then convert them to n_grams.

The default implementation works on a single sentences in the following order:

1. Decoding: the sentence according to given encoding (default 'utf-8')
2. Preprocessing: convert the sentence to lower case
3. Tokenizing: get single word tokens from the sentence (The default regexp selects tokens of 2 or more alphanumeric characters)
4. Stop word removal: remove the single word tokens from the above step which are present in stop words
5. N_gram creation: After stop word removal, the remaining tokens are then arranged in the required n_grams
6. Remove too rare or too common features: Remove words which have frequency greater than max_df or lower than min_df.

You need to handle all this if you want to pass a custom callable to the analyzer param in the TfidfVectorizer.

OR

You can extend the TfidfVectorizer class and only override the last 2 steps. Something like this:

from sklearn.feature_extraction.text import TfidfVectorizer
class NewTfidfVectorizer(TfidfVectorizer):
    def _word_ngrams(self, tokens, stop_words=None):

        # First get tokens without stop words
        tokens = super(TfidfVectorizer, self)._word_ngrams(tokens, None)
        if stop_words is not None:
            new_tokens=[]
            for token in tokens:
                split_words = token.split(' ')

                # Only check the first and last word for stop words
                if split_words[0] not in stop_words and split_words[-1] not in stop_words:
                    new_tokens.append(token)
            return new_tokens

        return tokens

Then, use it like:

vectorizer = NewTfidfVectorizer(stop_words='english', ngram_range=(3,3))
vectorizer.fit(data)