reproducer (version 0.5.3)

calculateMABias: calculateMABias

Description

The function simulates multiple five group families of either two-group or four-group experiments and estimates the power, individual estimate error, and the small sample bias obtained after synthesizing the analysis results obtained from the experiments in each family. The power is estimated as the percentage of families for which the overall mean of the five experiments was significantly different from zero. The experiment data may be one of four different type: Normal, Log-normal, Gamma or Laplace. The simulations can be repeated for different mean differences between the control mean and treatment mean depending on the parameter diff. The output is a table of values identifying the observed values of three effect sizes: Cliff's d, PHat and StdMD, estimate error and their related small sample bias and power for each set of simulated families. The synthesis method for all the effect sizes is based on calculating the overall mean and variance for experiments in each family and then using those values to calculate the overall effect size and its variance. This function supports the production of the values reported in data tables in the paper "Recommendations for Analyzing Small Sample Size Software Engineering Experiments" and its Supplementary Material.

Usage

calculateMABias(
  mean = 0,
  sd = 1,
  N,
  reps,
  diff = c(0.2, 0.5, 0.8),
  Experiments = 5,
  Expected.StdMD = c(0.2, 0.5, 0.8),
  Expected.PHat = c(0.556, 0.638, 0.714),
  type = "n",
  FourG = FALSE,
  seed = 223,
  StdAdj = 0,
  Blockmean = 0,
  StdExp = 0,
  MAMethod = "PM",
  alpha = 0.05
)

Value

Design. Specifies the type of experiment 2G or 4G, the sample distribution (n,l,g,lap), and whether variance heterogeneity was added (het)

BEIncluded. Specifies whether or not a block effect was introduced. Always set to "No" for two-group experiments.

GrpSize. Specifies the size of each group in the individual experiments.

Diff. The size of the difference between the control and treatment converted to an ordinal scale (Small, Medium, Large)

NPBias The relative difference between the average of the observed values of either Cliff's d or centralised PHat and the population value

StdMDBias. The relative difference between the average of the observed values of StdMDBias and the theoretical value

NPMdMRE The median of the absolute relative difference between the observed values of either Cliff's d or centralised PHat and the theoretical value for each experiment.

StdMDMdMRE The median of the absolute relative difference between the observed values of StdMD and the population value for each experiment.

ObsPHat. The average of the average Phat value found for each family in the set of simulations.

ObsCliffd. The average of the average Cliffd value found for each family in the set of simulations.

ObsStdES. The average of StdMD calculated for each family in the set of simulations.

PHatPower. The percentage of the simulations, for a specific mean difference, for which the overall Phat estimate was significantly different from zero at the nominated alpha level using one-sided tests.

CliffdPower. The percentage of the simulations, for a specific mean difference, for which the overall Cliff's d estimate was significantly different from zero at the nominated alpha level using one-sided tests.

StdMDPower. The percentage of the simulations, for a specific mean difference, for which the overall StdMD estimate was significantly different from zero at the nominated alpha level using one-sided tests.

Arguments

mean

This is the mean value of the control and treatment group(s) used in the simulations of each experiment of each family for simulations of a specified sample size (default 0).

sd

This is the standard deviation value of the control group(s) and treatment group(s) used in the simulations of each experiment of each family for simulations of a specified sample size (default 1).

N

This specifies the sample size per group that will be used in each set of simulations.

reps

The number of families simulated for each sample size.

diff

This specifies the difference between the control and treatment that will be used in each set of simulations. It must always have three values representing small, medium and large values (default c(0.2, 0.5, 0.8)).

Experiments

The number of experiments in each family (default 5).

Expected.StdMD

This defines the expected value of the overall average StdMD for each mean difference (default c(0.2,0.5,0.8)).

Expected.PHat

This defines the expected population value of the overall average Phat for each mean difference (default c(0.556,0.638,0.714)).

type

This specifies the distribution of the data samples that will be simulated. Options ae "n" for Normal, "l", for Log-normal,'g" for Gamma, "lap" for LaPlace (default "n").

FourG

If FourG is FALSE (default) the individual experiments in each family will be two-group experiments, otherwise the individual experiments will be four-group families.

seed

A seed for the simulations (default 123).

StdAdj

Used to introduce variance heterogeneity for Laplace and Normal samples (default 0).

Blockmean

Used to set a fixed block effect for four-group experiments (default 0).

StdExp

Used to introduce heterogeneity among families of experiments (default 0).

MAMethod

Not used (default "PM").

alpha

The significance level for statistical tests (default 0.05).

Author

Barbara Kitchenham and Lech Madeyski

Examples

# as.data.frame(calculateMABias(mean=0,sd=1,N=10,diff=c(0.2,0.5,0.8), Experiments=5,reps=10,
# Expected.StdMD=c(0.2,0.5,0.8), Expected.PHat=c(0.556,0.638,0.714), type="n",FourG=FALSE,
# seed= 123, StdAdj = 0, Blockmean=0, StdExp=0))
#  Design Blockmean GrpSize   Diff     NPBias  StdMDBias   NPMdMRE StdMDMdMRE ObsPHat ObsCliffd..
#1   2G_n        No      10  Small 0.09285714 0.02606704 0.8928571  1.0741432  0.5612    0.1224..
#2   2G_n        No      10 Medium 0.03768116 0.01740262 0.2391304  0.4171896  0.6432    0.2864..
#3   2G_n        No      10  Large 0.03738318 0.01523651 0.2009346  0.2490287  0.7220    0.4440..
#  PHatPower CliffdPower StdESPower
#1       0.2         0.2        0.3
#2       0.7         0.7        0.7
#3       1.0         1.0        1.0
as.data.frame(calculateMABias(mean=0,sd=1,N=10,diff=c(0.2,0.5,0.8), Experiments=5,reps=4,
 Expected.StdMD=c(0.2,0.5,0.8), Expected.PHat=c(0.556,0.638,0.714), type="n",FourG=TRUE,
 seed= 123,StdAdj = 0.5,Blockmean=0.5,StdExp=0))
 #Results for reps=10
#    Design Blockmean GrpSize   Diff     NPBias  StdMDBias   NPMdMRE StdMDMdMRE ObsPHat ObsClif..
#1 4G_n_het       Yes      10  Small -0.1321429 -0.1372277 0.6696429  0.4698935  0.5486  0.0972..
#2 4G_n_het       Yes      10 Medium -0.1869565 -0.1882479 0.2318841  0.1472392  0.6122  0.2244..
#3 4G_n_het       Yes      10  Large -0.1864486 -0.2010029 0.1612150  0.1531253  0.6741  0.3482..
#  PHatPower CliffdPower StdESPower
#1       0.4         0.4        0.4
#2       0.9         0.9        0.8
#3       1.0         1.0        1.0