What is the best way to obtain the optimal number of topics for a LDA-Model using Gensim?

Question

I am trying to obtain the optimal number of topics for an LDA-model within Gensim. One method I found is to calculate the log likelihood for each model and compare each agai

Answer 1

Although I cannot comment on Gensim in particular I can weigh in with some general advice for optimising your topics.

As you stated, using log likelihood is one method. Another option is to keep a set of documents held out from the model generation process and infer topics over them when the model is complete and check if it makes sense.

A completely different method you could try is a hierarchical Dirichlet process, this method can find the number of topics in the corpus dynamically without being specified.

There are many papers on how to best specify parameters and evaluate your topic model, depending on your experience level these may or may not be good for you:

Rethinking LDA: Why Priors Matter, Wallach, H.M., Mimno, D. and McCallum, A.

Evaluation Methods for Topic Models, Wallach H.M., Murray, I., Salakhutdinov, R. and Mimno, D.

Also, here is the paper about the hierarchical Dirichlet process:

Hierarchical Dirichlet Processes, Teh, Y.W., Jordan, M.I., Beal, M.J. and Blei, D.M.