gensim.LDAMulticore throwing exception:

问题:

I am running LDAMulticore from the python gensim library, and the script cannot seem to create more than one thread. Here is the error:

  Traceback (most recent call last):   File "/usr/lib64/python2.7/multiprocessing/process.py", line 258, in _bootstrap     self.run()   File "/usr/lib64/python2.7/multiprocessing/process.py", line 114, in run     self._target(*self._args, **self._kwargs)   File "/usr/lib64/python2.7/multiprocessing/pool.py", line 97, in worker     initializer(*initargs)   File "/usr/lib64/python2.7/site-packages/gensim/models/ldamulticore.py", line 333, in worker_e_step     worker_lda.do_estep(chunk)  # TODO: auto-tune alpha?   File "/usr/lib64/python2.7/site-packages/gensim/models/ldamodel.py", line 725, in do_estep     gamma, sstats = self.inference(chunk, collect_sstats=True)   File "/usr/lib64/python2.7/site-packages/gensim/models/ldamodel.py", line 655, in inference     ids = [int(idx) for idx, _ in doc] TypeError: 'int' object is not iterable Exception in thread Thread-1: Traceback (most recent call last):   File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner     self.run()   File "/usr/lib64/python2.7/threading.py", line 765, in run     self.__target(*self.__args, **self.__kwargs)   File "/usr/lib64/python2.7/multiprocessing/pool.py", line 325, in _handle_workers     pool._maintain_pool()   File "/usr/lib64/python2.7/multiprocessing/pool.py", line 229, in _maintain_pool     self._repopulate_pool()   File "/usr/lib64/python2.7/multiprocessing/pool.py", line 222, in _repopulate_pool     w.start()   File "/usr/lib64/python2.7/multiprocessing/process.py", line 130, in start     self._popen = Popen(self)   File "/usr/lib64/python2.7/multiprocessing/forking.py", line 121, in __init__     self.pid = os.fork() OSError: [Errno 12] Cannot allocate memory 

I'm creating my LDA model like this:

ldamodel = LdaMulticore(corpus, num_topics=50, id2word = dictionary, workers=3) 

I have actually asked another question about this script, so the full script can be found here:

Gensim LDA Multicore Python script runs much too slow

If it's relevant, I'm running this on a CentOS server. Let me know if I should include any other information.

Any help is appreciated!

回答1:

OSError: [Errno 12] Cannot allocate memory sounds like you are running out of RAM.

Check your available free memory and swap.

You can try to to reduce the number of threads with the workers parameter or the number of documents to be used in each training chunk with the chunksize parameter.

文章来源: gensim.LDAMulticore throwing exception: