MetabolSSMF (version 0.1.0)

gap: Gap statistic algorithm.

Description

Estimating the number of prototypes/clusters in a data set using the gap statistic.

Usage

gap(
  data,
  rss,
  meth = c("kmeans", "uniform", "dirichlet", "nmf"),
  itr = 50,
  lr = 0.01,
  ncore = 2
)

Value

gap Gap value vector.

optimal.k The optimal number of prototypes/clusters.

standard.error Standard error vector.

Arguments

data

Data matrix or data frame.

rss

Numeric vector, residual sum of squares from ssmf model using the number of clusters \(1,2, \ldots, k\).

meth

Character, specification of method to initialise the \(W\) and \(H\) matrix, see 'method' in init( ).

itr

Integer, number of Monte Carlo samples.

lr

Optimisation learning rate in ssmf().

ncore

The number of cores to use for parallel execution.

Author

Wenxuan Liu

Details

This gap statistic selects the biggest difference between the original residual sum of squares (RSS) and the RSS under an appropriate null reference distribution of the data, which is defined to be $$\mathrm{Gap}(k) = \frac{1}{B} \sum_{b=1}^{B} \log(\mathrm{RSS}^*_{kb}) - \log(\mathrm{RSS}_{k}),$$

where \(B\) is the number of samples from the reference distribution; \(\mathrm{RSS}^*_{kb}\) is the residual sum of squares of the \(b^th\) sample from the reference distribution fitted in the SSMF model model using \(k\) clusters; \(RSS_{k}\) is the residual sum of squares for the original data \(X\) fitted the model using the same \(k\). The estimated gap suggests the number of prototypes/clusters (\(\hat{k}\)) using

$$\hat{k} = \mathrm{smallest} \ k \ \mathrm{such \ that} \ \mathrm{Gap}(k) \geq \mathrm{Gap}(k+1) - s_{k+1},$$

where \(s_{k+1}\) is standard error that is defined as

$$s_{k+1}=sd_k \sqrt{1+\frac{1}{B}},$$

and \(sd_k\) is the standard deviation:

$$sd_k=\sqrt{ \frac{1}{B} \sum_{b} [\log(\mathrm{RSS}^*_{kb})-\frac{1}{B} \sum_{b} \log(\mathrm{RSS}^*_{kb})]^2}.$$

References

Tibshirani, R., Walther, G., & Hastie, T. (2001). Estimating the Number of Clusters in a Data Set via the Gap Statistic. Journal of the Royal Statistical Society. Series B (Statistical Methodology), 63(2), 411–423. <doi:10.1111/1467-9868.00293>

Examples

# example code
# \donttest{
data <- SimulatedDataset

k <- 6

rss <- rep(NA, k)
for(i in 1:k){
  rss[i] <- ssmf(data = data, k = i)$SSE
}

gap(data = data, rss = rss)
# }