SignalY (version 1.1.1)

compute_entropy: Compute Shannon Entropy

Description

Calculates the Shannon entropy of a probability distribution or, when applied to loadings, the entropy of the squared normalized loadings. High entropy indicates diffuse/uniform distribution (systemic noise), while low entropy indicates concentrated structure.

Usage

compute_entropy(x, base = 2, normalize = FALSE)

Value

Numeric scalar representing entropy value.

Arguments

x

Numeric vector. Will be squared and normalized to form a probability distribution.

base

Base of the logarithm. Default is 2 (bits).

normalize

Logical. If TRUE, returns normalized entropy (0 to 1 scale).

Interpretation in Signal Analysis

In the context of latent structure extraction:

• High entropy (near maximum): Suggests "maximum entropy systemic stochasticity" - the component captures diffuse, undifferentiated movement across all variables (akin to Brownian motion).

• Low entropy: Suggests "differentiated latent structure" - the component is driven by a subset of variables, indicating meaningful structural relationships.

Details

The Shannon entropy is defined as: $$H(p) = -\sum_{i} p_i \log(p_i)$$

where \(p_i\) are the probabilities. For factor loadings, we use squared normalized loadings as the probability distribution: $$p_i = \lambda_i^2 / \sum_j \lambda_j^2$$

This measures the concentration of explanatory power across variables. Maximum entropy occurs when all loadings are equal (diffuse structure); minimum entropy occurs when a single variable dominates (concentrated structure).

Examples

uniform_loadings <- rep(1, 10)
compute_entropy(uniform_loadings, normalize = TRUE)

concentrated_loadings <- c(10, rep(0.1, 9))
compute_entropy(concentrated_loadings, normalize = TRUE)