gensim

gensim doc2vec “intersect_word2vec_format” command

由 大城市里の小女人 提交于 2019-12-20 03:55:16
问题 Just reading through the doc2vec commands on the gensim page. I am curious about the command"intersect_word2vec_format" . My understanding of this command is it lets me inject vector values from a pretrained word2vec model into my doc2vec model and then train my doc2vec model using the pretrained word2vec values rather than generating the word vector values from my document corpus. The result is that I get a more accurate doc2vec model because I am using pretrained w2v values which was

Error in extracting phrases using Gensim

由 我只是一个虾纸丫 提交于 2019-12-20 03:50:51
问题 I am trying to get the bigrams in the sentences using Phrases in Gensim as follows. from gensim.models import Phrases from gensim.models.phrases import Phraser documents = ["the mayor of new york was there", "machine learning can be useful sometimes","new york mayor was present"] sentence_stream = [doc.split(" ") for doc in documents] #print(sentence_stream) bigram = Phrases(sentence_stream, min_count=1, threshold=2, delimiter=b' ') bigram_phraser = Phraser(bigram) for sent in sentence_stream

How to fix 'C extension not loaded, training will be slow. Install a C compiler and reinstall gensim for fast training.'

由 家住魔仙堡 提交于 2019-12-20 01:06:46
问题 I'm using the library node2vec, which is based on gensim word2vec model to encode nodes in an embedding space, but when i want to fit the word2vec object I get this warning: C:\Users\lenovo\Anaconda3\lib\site-packages\gensim\models\base_any2vec.py:743: UserWarning: C extension not loaded, training will be slow. Install a C compiler and reinstall gensim for fast training. Can any one help me to fix this issue please ? 回答1: gensim relies on extension modules that need to be compiled. Both

Using word2vec to classify words in categories

由 别来无恙 提交于 2019-12-18 11:31:31
问题 BACKGROUND I have vectors with some sample data and each vector has a category name (Places,Colors,Names). ['john','jay','dan','nathan','bob'] -> 'Names' ['yellow', 'red','green'] -> 'Colors' ['tokyo','bejing','washington','mumbai'] -> 'Places' My objective is to train a model that take a new input string and predict which category it belongs to. For example if a new input is "purple" then I should be able to predict 'Colors' as the correct category. If the new input is "Calgary" it should

gensim LdaMulticore not multiprocessing?

由 混江龙づ霸主 提交于 2019-12-18 06:07:33
问题 When I run gensim's LdaMulticore model on a machine with 12 cores, using: lda = LdaMulticore(corpus, num_topics=64, workers=10) I get a logging message that says using serial LDA version on this node A few lines later, I see another loging message that says training LDA model using 10 processes When I run top, I see 11 python processes have been spawned, but 9 are sleeping, I.e. only one worker is active. The machine has 24 cores, and is not overwhelmed by any means. Why isn't LdaMulticore

gensim LdaMulticore not multiprocessing?

由 一笑奈何 提交于 2019-12-18 06:07:33
问题 When I run gensim's LdaMulticore model on a machine with 12 cores, using: lda = LdaMulticore(corpus, num_topics=64, workers=10) I get a logging message that says using serial LDA version on this node A few lines later, I see another loging message that says training LDA model using 10 processes When I run top, I see 11 python processes have been spawned, but 9 are sleeping, I.e. only one worker is active. The machine has 24 cores, and is not overwhelmed by any means. Why isn't LdaMulticore

How to use Gensim doc2vec with pre-trained word vectors?

由 谁说我不能喝 提交于 2019-12-17 17:24:36
问题 I recently came across the doc2vec addition to Gensim. How can I use pre-trained word vectors (e.g. found in word2vec original website) with doc2vec? Or is doc2vec getting the word vectors from the same sentences it uses for paragraph-vector training? Thanks. 回答1: Note that the "DBOW" ( dm=0 ) training mode doesn't require or even create word-vectors as part of the training. It merely learns document vectors that are good at predicting each word in turn (much like the word2vec skip-gram

PyTorch / Gensim - How to load pre-trained word embeddings

由 一笑奈何 提交于 2019-12-17 10:32:39
问题 I want to load a pre-trained word2vec embedding with gensim into a PyTorch embedding layer. So my question is, how do I get the embedding weights loaded by gensim into the PyTorch embedding layer. Thanks in Advance! 回答1: I just wanted to report my findings about loading a gensim embedding with PyTorch. Solution for PyTorch 0.4.0 and newer: From v0.4.0 there is a new function from_pretrained() which makes loading an embedding very comfortable. Here is an example from the documentation. >> #

LDA model generates different topics everytime i train on the same corpus

由 試著忘記壹切 提交于 2019-12-17 09:34:11
问题 I am using python gensim to train an Latent Dirichlet Allocation (LDA) model from a small corpus of 231 sentences. However, each time i repeat the process, it generates different topics. Why does the same LDA parameters and corpus generate different topics everytime? And how do i stabilize the topic generation? I'm using this corpus (http://pastebin.com/WptkKVF0) and this list of stopwords (http://pastebin.com/LL7dqLcj) and here's my code: from gensim import corpora, models, similarities from

自然语言处理库—Gensim之Word2vec

由 倖福魔咒の 提交于 2019-12-15 10:09:12
1. gensim概述 Gensim（http://pypi.python.org/pypi/gensim）是一款开源的第三方Python工具包，用于从原始的非结构化的文本中，无监督地学习到文本隐层的主题向量表达。 主要用于主题建模和文档相似性处理，它支持包括TF-IDF，LSA，LDA，和word2vec在内的多种主题模型算法。Gensim在诸如获取单词的词向量等任务中非常有用。 使用Gensim训练Word2vec十分方便，训练步骤如下： 1）将语料库预处理：一行一个文档或句子，将文档或句子分词（以空格分割，英文可以不用分词，英文单词之间已经由空格分割，中文预料需要使用分词工具进行分词，常见的分词工具有StandNLP、ICTCLAS、Ansj、FudanNLP、HanLP、结巴分词等）； 2）将原始的训练语料转化成一个sentence的迭代器，每一次迭代返回的sentence是一个word（utf8格式）的列表。可以使用Gensim中word2vec.py中的LineSentence()方法实现； 3）将上面处理的结果输入Gensim内建的word2vec对象进行训练即可： from gensim . models import Word2Vec sentences = word2vec . LineSentence ( './in_the_name_of_people
订阅 gensim