divergence: Optimize the number of topics for LDA

Description

divergence() computes the regularized topic divergence scores to help users to find the optimal number of topics for LDA.

Usage

divergence(
  x,
  min_size = 0.01,
  select = NULL,
  regularize = TRUE,
  newdata = NULL,
  ...
)

Value

Returns a singple numeric value.

Arguments

x: a LDA model fitted by textmodel_seededlda() or textmodel_lda().
min_size: the minimum size of topics for regularized topic divergence. Ignored when regularize = FALSE.
select: names of topics for which the divergence is computed.
regularize: if TRUE, returns the regularized divergence.
newdata: if provided, theta and phi are estimated through fresh Gibbs sampling.
...: additional arguments passed to textmodel_lda.

Details

divergence() computes the average Jensen-Shannon divergence between all the pairs of topic vectors in x$phi. The divergence score maximizes when the chosen number of topic k is optimal (Deveaud et al., 2014). The regularized divergence penalizes topics smaller than min_size to avoid fragmentation (Watanabe & Baturo, forthcoming).

References

Deveaud, Romain et al. (2014). "Accurate and Effective Latent Concept Modeling for Ad Hoc Information Retrieval". doi:10.3166/DN.17.1.61-84. Document Numérique.