PublicationBiasBenchmark (version 0.1.3)

measures: Performance Measures and Monte Carlo Standard Errors

Description

A comprehensive set of functions for computing performance measures and their Monte Carlo Standard Errors (MCSE) for simulation studies. All functions are based on definitions from Table 3 in siepe2024simulation;textualPublicationBiasBenchmark. Winkler interval score is defined in winkler1972decision;textualPublicationBiasBenchmark. Positive and negative likelihood ratios are defined in huang2023relative;textualPublicationBiasBenchmark and deeks2004diagnostic;textualPublicationBiasBenchmark. Also see morris2019using;textualPublicationBiasBenchmark for additional details. Bias and relative bias were modified to account for possibly different true values across repetitions.

Usage

bias(theta_hat, theta)
bias_mcse(theta_hat)
relative_bias(theta_hat, theta)
relative_bias_mcse(theta_hat, theta)
mse(theta_hat, theta)
mse_mcse(theta_hat, theta)
rmse(theta_hat, theta)
rmse_mcse(theta_hat, theta)
empirical_variance(theta_hat)
empirical_variance_mcse(theta_hat)
empirical_se(theta_hat)
empirical_se_mcse(theta_hat)
coverage(ci_lower, ci_upper, theta)
coverage_mcse(ci_lower, ci_upper, theta)
power(test_rejects_h0)
power_mcse(test_rejects_h0)
mean_ci_width(ci_upper, ci_lower)
mean_ci_width_mcse(ci_upper, ci_lower)
mean_generic_statistic(G)
mean_generic_statistic_mcse(G)
positive_likelihood_ratio(tp, fp, fn, tn)
positive_likelihood_ratio_mcse(tp, fp, fn, tn)
negative_likelihood_ratio(tp, fp, fn, tn)
negative_likelihood_ratio_mcse(tp, fp, fn, tn)
interval_score(ci_lower, ci_upper, theta, alpha = 0.05)
interval_score_mcse(ci_lower, ci_upper, theta, alpha = 0.05)

Value

Each metric function returns a numeric value representing the performance measure. Each MCSE function returns a numeric value representing the Monte Carlo standard error.

Arguments

theta_hat

Vector of parameter estimates from simulations

theta

True parameter value

ci_lower

Vector of lower confidence interval bounds

ci_upper

Vector of upper confidence interval bounds

test_rejects_h0

Logical vector indicating whether statistical tests reject the null hypothesis

G

Vector of generic statistics from simulations

tp

Numeric with the count of true positive hypothesis tests

fp

Numeric with the count of false positive hypothesis tests

fn

Numeric with the count of false negative hypothesis tests

tn

Numeric with the count of true negative hypothesis tests

alpha

Numeric indicating the 1 - coverage level for interval_score calculation

Details

The package provides the following performance measures and their corresponding MCSE functions:

• bias(theta_hat, theta): Bias estimate

• relative_bias(theta_hat, theta): Relative bias estimate

• mse(theta_hat, theta): Mean Square Error

• rmse(theta_hat, theta): Root Mean Square Error

• empirical_variance(theta_hat): Empirical variance

• empirical_se(theta_hat): Empirical standard error

• coverage(ci_lower, ci_upper, theta): Coverage probability

• mean_ci_width(ci_upper, ci_lower): Mean confidence interval width

• interval_score(ci_lower, ci_upper, theta, alpha): interval_score

• power(test_rejects_h0): Statistical power

• positive_likelihood_ratio(tp, fp, fn, tn): Log positive likelihood ratio

• negative_likelihood_ratio(tp, fp, fn, tn): Log negative likelihood ratio

• mean_generic_statistic(G): Mean of any generic statistic

References

Examples

# Generate some example data
set.seed(123)
theta_true <- 0.5
theta_estimates <- rnorm(1000, mean = theta_true, sd = 0.1)

# Compute bias and its MCSE
bias_est <- bias(theta_estimates, theta_true)
bias_se <- bias_mcse(theta_estimates)

# Compute MSE and its MCSE
mse_est <- mse(theta_estimates, theta_true)
mse_se <- mse_mcse(theta_estimates, theta_true)

# Example with coverage
ci_lower <- theta_estimates - 1.96 * 0.1
ci_upper <- theta_estimates + 1.96 * 0.1
coverage_est <- coverage(ci_lower, ci_upper, theta_true)
coverage_se <- coverage_mcse(ci_lower, ci_upper, theta_true)