gensim

I have X as a csr_matrix that I obtained using scikit's tfidf vectorizer, and y which is an array My plan is to create features using LDA, however, I failed to find how to initialize a gensim's corpus variable with X as a csr_matrix. In other words, I don't want to download a corpus as shown in gensim's documentation nor convert X to a dense matrix, since it would consume a lot of memory and the computer could hang. In short, my questions are the following, How do you initialize a gensim corpus given that I have a csr_matrix (sparse) representing the whole corpus? How do you use LDA to extract

I am using pre-trained Google news dataset for getting word vectors by using Gensim library in python model = Word2Vec.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True) After loading the model I am converting training reviews sentence words into vectors #reading all sentences from training file with open('restaurantSentences', 'r') as infile: x_train = infile.readlines() #cleaning sentences x_train = [review_to_wordlist(review,remove_stopwords=True) for review in x_train] train_vecs = np.concatenate([buildWordVector(z, n_dim) for z in x_train]) During word2Vec process i

BACKGROUND I have vectors with some sample data and each vector has a category name (Places,Colors,Names). ['john','jay','dan','nathan','bob'] -> 'Names' ['yellow', 'red','green'] -> 'Colors' ['tokyo','bejing','washington','mumbai'] -> 'Places' My objective is to train a model that take a new input string and predict which category it belongs to. For example if a new input is "purple" then I should be able to predict 'Colors' as the correct category. If the new input is "Calgary" it should predict 'Places' as the correct category. APPROACH I did some research and came across Word2vec . This

For preprocessing the corpus I was planing to extarct common phrases from the corpus, for this I tried using Phrases model in gensim, I tried below code but it's not giving me desired output. My code from gensim.models import Phrases documents = ["the mayor of new york was there", "machine learning can be useful sometimes"] sentence_stream = [doc.split(" ") for doc in documents] bigram = Phrases(sentence_stream) sent = [u'the', u'mayor', u'of', u'new', u'york', u'was', u'there'] print(bigram[sent]) Output [u'the', u'mayor', u'of', u'new', u'york', u'was', u'there'] But it should come as [u'the

I've got a trained LDA model and I want to calculate the similarity score between two documents from the corpus I trained my model on. After studying all the Gensim tutorials and functions, I still can't get my head around it. Can somebody give me a hint? Thanks! Don't know if this'll help but, I managed to attain successful results on document matching and similarities when using the actual document as a query. dictionary = corpora.Dictionary.load('dictionary.dict') corpus = corpora.MmCorpus("corpus.mm") lda = models.LdaModel.load("model.lda") #result from running online lda (training) index

When I run gensim's LdaMulticore model on a machine with 12 cores, using: lda = LdaMulticore(corpus, num_topics=64, workers=10) I get a logging message that says using serial LDA version on this node A few lines later, I see another loging message that says training LDA model using 10 processes When I run top, I see 11 python processes have been spawned, but 9 are sleeping, I.e. only one worker is active. The machine has 24 cores, and is not overwhelmed by any means. Why isn't LdaMulticore running in parallel mode? First, make sure you have installed a fast BLAS library , because most of the