weird (version 2.0.0)

surprisals.numeric: Surprisals and surprisal probabilities computed from data

Description

A surprisal is given by \(s = -\log f(y)\) where \(f\) is the density or probability mass function of the estimated or assumed distribution, and \(y\) is an observation. This is returned by surprisals(). A surprisal probability is the probability of a surprisal at least as extreme as \(s\). This is returned by surprisals_prob()

Usage

# S3 method for numeric
surprisals(object, distribution = dist_kde(object, ...), loo = FALSE, ...)
# S3 method for matrix
surprisals(object, distribution = dist_kde(object, ...), loo = FALSE, ...)
# S3 method for data.frame
surprisals(object, distribution = dist_kde(object, ...), loo = FALSE, ...)
# S3 method for numeric
surprisals_prob(
  object,
  approximation = c("none", "gpd", "rank"),
  threshold_probability = 0.1,
  distribution = dist_kde(object, ...),
  loo = FALSE,
  ...
)
# S3 method for matrix
surprisals_prob(
  object,
  approximation = c("none", "gpd", "rank"),
  threshold_probability = 0.1,
  distribution = dist_kde(object, ...),
  loo = FALSE,
  ...
)
# S3 method for data.frame
surprisals_prob(
  object,
  approximation = c("none", "gpd", "rank"),
  threshold_probability = 0.1,
  distribution = dist_kde(object, ...),
  loo = FALSE,
  ...
)

Value

A numerical vector containing the surprisals or surprisal probabilities.

Arguments

object

A numerical data set (either a vector, matrix, or a data.frame containing only numerical columns).

distribution

A distribution object. By default, a kernel density estimate is computed from the data object.

loo

Should leave-one-out surprisals be computed?

...

Other arguments are passed to the appropriate method.

approximation

Character string specifying the method to use in computing the surprisal probabilities. See Details below. For a multivariate data set, it needs to be set to either "gpd" or "rank".

threshold_probability

Probability threshold when computing the GPD approximation. This is the probability below which the GPD is fitted. Only used if approximation = "gpd".

Author

Rob J Hyndman

Details

The surprisal probabilities may be computed in three different ways.

1. When approximation = "none" (the default), the surprisal probabilities are computed using the same distribution that was used to compute the surprisal values. Under this option, surprisal probabilities are equal to 1 minus the coverage probability of the largest HDR that contains each value. Surprisal probabilities smaller than 1e-6 are returned as 1e-6.

2. When approximation = "gdp", the surprisal probabilities are computed using a Generalized Pareto Distribution fitted to the most extreme surprisal values (those with probability less than threshold_probability). For surprisal probabilities greater than threshold_probability, the value of threshold_probability is returned. Under this option, the distribution is used for computing the surprisal values but not for determining their probabilities. Due to extreme value theory, the resulting probabilities should be relatively insensitive to the distribution used in computing the surprisal values.

3. When approximation = "rank", the surprisal probability of each observation is estimated using the proportion of observations with greater surprisal values; i.e., 1 - rank(s)/n where rank(s) is the rank of the surprisal value s among all surprisal values, and n is the number of observations. This is a nonparametric approach that is also insensitive to the distribution used in computing the surprisal values.

References

Rob J Hyndman (2026) "That's weird: Anomaly detection using R", Chapter 6, https://OTexts.com/weird/.

See Also

dist_kde

Examples

# Univariate data
tibble(
  y = c(5, rnorm(49)),
  p_kde = surprisals_prob(y, loo = TRUE),
  p_normal = surprisals_prob(y, distribution = dist_normal()),
  p_zscore = 2 * (1 - pnorm(abs(y)))
)
tibble(
  y = n01$v1,
  prob1 = surprisals_prob(y),
  prob2 = surprisals_prob(y, loo = TRUE),
  prob3 = surprisals_prob(y, distribution = dist_normal()),
  prob4 = surprisals_prob(y, distribution = dist_normal(), approximation = "gpd")
) |>
  arrange(prob1)
# Bivariate data
tibble(
  x = rnorm(50),
  y = c(5, rnorm(49)),
  prob = surprisals_prob(cbind(x, y), approximation = "gpd")
)
oldfaithful |>
  mutate(
    s = surprisals(cbind(duration, waiting), loo = TRUE),
    p = surprisals_prob(cbind(duration, waiting), loo = TRUE, approximation = "gpd")
  ) |>
  arrange(p)