Parameter Testing

LDA Parameters Testing

Before training a Latent Dirichlet Allocation (LDA) model on the entire, large Wikipedia corpus, it would be helpful to do some small-scale testing of some of the parameter settings available in Gensim’s LDA implementation.

I did this small-scale testing on a portion of Wikipedia with only 15,151 documents. I downloaded it from here:

ftp://ftpmirror.your.org/pub/wikimedia/dumps/enwiki/20170701/enwiki-20170701-pages-articles1.xml-p10p30302.bz2

Coherence

First, we need to determine how to measure how good a model is. A good topic model will produce a set of topics that we humans would perceive as sensible. More concretely, it would make sense to us for the topic “Fantasy” to include Tolkien’s The Lord of the Rings and J. K. Rowling’s Harry Potter series, but not Lee’s To Kill a Mockingbird.

Fortunately, the hard work has already been done for us. Gensim provides several measurements of the coherence of topics, and Roder, Both, and Hinneburg (2015) determined that the Cv measurement provides the highest correlations with humans’ judgments of good (and bad) topics. The Cv measurement also takes the longest to calculate, but we’ll endure that for the sake of high quality results in the end.

LDA Hyperparameters

The most important parameters are the Bayesian prior hyperparameters alpha and beta (called ‘eta’ in Gensim). Both hyperparameters represent our prior beliefs about the corpus before we train our model. A high alpha generally makes the combination of topics associated with each document more uniform across documents; that is, the documents become more similar to each other regarding their topics. A high beta generally makes each topic contain a more uniform mix of words.

In Gensim, both alpha and beta can be set to ‘symmetric’, ‘asymmetric’, or ‘auto’. The default for both is a symmetric distribution of 1 divided by the number of topics. For ‘auto’, the model automatically learns the best values for the hyperparameters as it is trained on more and more data. The trade-off is that ‘auto’ takes longer to compute than ‘symmetric’ or ‘asymmetric’.

Beta

Below are some excerpts from a table showing the results of training multiple LDA models with different parameters on our small portion of Wikipedia. Because we’re training on a small corpus, we’re not going to run only one pass through the corpus. We’ll run many — 15, in this case. That’s probably far more than enough.

Each pair of models compares running beta (labeled as ‘eta’ in the table) as ‘auto’ versus running it as the default, which is ‘symmetric’. The last column is our Cv coherence measurement. For each pair of models, setting beta to ‘symmetric’ improves the coherence. And as a bonus, it generally takes less time to run, so we’ll keep beta on its default.

Alpha

Now let’s test alpha:

Setting alpha to ‘auto’ improves the coherences.

Other Parameters

I looked at parameters other than alpha and beta, too. The number of topics was the most important remaining parameter, but testing it on such a small corpus probably wouldn’t reveal much about what its value should be on the entire Wikipedia corpus. Testing it was more useful for gaining a feel for how the models and their coherences behave as the topic number changes.

Here are all the models, in case you’d like to see other how the models behave when other parameters are changed: