reproducer (version 0.5.3)

calculateMAType1Error: calculateMAType1Error

Description

The function simulates multiple five group families of either two-group or four-group experiments and estimates the Type1 Error rate obtained after synthesizing the analysis results obtained from the experiments in each family. The Type1 Error is estimated as the percentage of families for which the overall mean of the five experiments was significantly different from zero. The experiment data may be one of four different type: Normal, Log-normal, Gamma or Laplace. The simulations can be repeated for different sample sizes depending on the parameter N. The output is a table of values identifying three observed effect size estimates (Cliff's d, PHat and StdMD) and their related type 1 error rates for each set of simulated families. The synthesis method for all three effect sizes is based on calculating the overall mean and variance for the family of experiments, and then using those values to calculate the effect size variance and its variance. This function supports the production of the values reported in data tables in the paper "Recommendations for Analyzing Small Sample Size Software Engineering Experiments" and its Supplementary Material.

Usage

calculateMAType1Error(
  mean = 0,
  sd = 1,
  N = c(5, 10, 15, 20, 30, 40),
  reps,
  type = "n",
  seed = 123,
  Experiments = 5,
  FourG = FALSE,
  StdAdj = 0,
  Blockmean = 0,
  BlockStdAdj = 0,
  StdExp = 0,
  MAMethod = "PM",
  alpha = 0.05
)

Value

Design. Specifies the type of experiment 2G or 4G, the sample distribution (n,l,g,lap), and whether variance heterogeneity was added (het)

BEIncluded. Specifies whether or not a block effect was introduced. Always set to "No" for two-group experiments.

GrpSize. Specifies the size of each group in the individual experiments.

ObsPHat. The average of the average Phat value found for each family in the set of simulations.

ObsCliffd. The average of the average Cliffd value found for each family in the set of simulations.

ObsStdES. The average of StdMD calculated for each family in the set of simulations.

PHatType1ER. The percentage of the simulations, for a specific group size, for which the overall Phat estimate was significantly different from zero at the nominated alpha level.

CliffdType1ER. The percentage of the simulations, for a specific group size, for which the overall Cliff's d estimate was significantly different from zero at the nominated alpha level.

StdMDType1ER. The percentage of the simulations, for a specific group size, for which the overall StdMD estimate was significantly different from zero at the nominated alpha level.

Arguments

mean

This is the mean value of the control and treatment group(s) used in the simulations of each experiment of each family for simulations of a specified sample size (default 0).

sd

This is the standard deviation value of the control group(s) and treatment group(s) used in the simulations of each experiment of each family for simulations of a specified sample size (default 1).

N

This specifies the sample sizes per group that will be used in each set of simulations (default c(5,10,15,20,30,40)).

reps

The number of families simulated for each sample size.

type

This specifies the distribution of the data samples that will be simulated. Options ae "n" for Normal, "l", for Log-normal,'g" for Gamma, "lap" for LaPlace (default "n").

seed

A seed for the simulations (default 123).

Experiments

The number of experiments in each family (default 5).

FourG

If FourG is FALSE the individual experiments in each family will be two-group experiments, otherwise the individual experiments will be four-group families (default FALSE).

StdAdj

Used to introduce variance heterogeneity for Laplace and Normal samples (default 0).

Blockmean

Used to set a fixed block effect for four-group experiments (default 0).

BlockStdAdj

Not used (default 0).

StdExp

Used to introduce heterogeneity among families of experiments (default 0).

MAMethod

Not used (default "PM").

alpha

The significance level for statistical tests (default 0.05).

Author

Barbara Kitchenham and Lech Madeyski

Examples

# as.data.frame(calculateMAType1Error(mean=0,sd=1,N=c(5,10),reps=10,type="n",Experiments=5,
#  FourG=FALSE,StdAdj=0,Blockmean=0,seed=123))
#  Design BEIncluded GrpSize ObsPHat ObsCliffd     ObsStdES PHatType1ER CliffdType1ER StdMDType1ER
#1   2G_n         No       5  0.4848   -0.0304 -0.054156883           0             0          0.0
#2   2G_n         No      10  0.5036    0.0072  0.002888142           0             0          0.1

#as.data.frame(calculateMAType1Error(mean=0,sd=1,N=c(5,10),reps=10,type="l",Experiments=5,
#  FourG=FALSE,StdAdj=0,Blockmean=0,seed=123))
#   Design BEIncluded GrpSize ObsPHat ObsCliffd    ObsStdES PHatType1ER CliffdType1ER StdMDType1ER
#1   2G_l         No       5  0.4848   -0.0304 -0.02789656           0             0          0.0
#2   2G_l         No      10  0.5036    0.0072  0.06473696           0             0          0.2

#as.data.frame(calculateMAType1Error(mean=0,sd=1,N=c(5,10),reps=10,type="n",Experiments=5,
#  FourG=TRUE,StdAdj=0.5,Blockmean=0.5,seed=123))
#   Design BEIncluded GrpSize ObsPHat ObsCliffd   ObsStdES PHatType1ER CliffdType1ER StdMDType1ER
#  1 4G_n_het        Yes       5  0.5108    0.0216 0.01361820           0             0          0.1
#  2 4G_n_het        Yes      10  0.5069    0.0138 0.01700672           0             0          0.0

as.data.frame(calculateMAType1Error(mean=0,sd=1,N=c(5,10),reps=5,type="l",Experiments=5,
 FourG=TRUE,StdAdj=0,Blockmean=0.5,seed=123))
 #Results for reps=10
#   Design BEIncluded GrpSize ObsPHat ObsCliffd   ObsStdES PHatType1ER CliffdType1ER StdMDType1ER
#1   4G_l        Yes       5  0.5108    0.0216 0.07578257           0             0          0.2
#2   4G_l        Yes      10  0.5072    0.0144 0.04839936           0             0          0.0