Learn R Programming
greybox (version 0.3.3)

rmc: RMC test

Description

RMC stands for "Regression for Methods Comparison". This is a parametric test for the comparison of means of several distributions This test is a parametric counterpart of nemenyi / MCB test (Demsar, 2006) and uses asymptotic properties of regression models. It relies on distributional assumptions about the provided data. For instance, if the mean forecast errors are used, then it is safe to assume that the regression model constructed on them will have normally distributed residuals.

Usage

rmc(data, distribution = c("dnorm", "dfnorm", "dchisq"), level = 0.95,
  style = c("mcb", "lines"), select = NULL, plot = TRUE, ...)

Arguments

data

Matrix or data frame with observations in rows and variables in columns.

distribution

Type of the distribution to use. If this is a clear forecast error, then "dnorm" is appropriate, leading to a simple Gausian linear regression. "dfnorm" would lead to a alm model with folded normal distribution. Finally, "dchisq" would lead to the alm with Chi squared distribution. This value is passed to alm() function.

level

The width of the confidence interval. Default is 0.95.

style

What style of plot to use after the calculations. This can be either "MCB" style or "Vertical lines" one.

select

What column of data to highlight on the plot. If NULL, then the method with the lowest value is selected.

plot

If TRUE then the graph is produced after the calculations. You can also use plot method on the produced object in order to get the same effect.

...

Other parameters passed to plot function

Value

If plot=TRUE, then the function plots the results after all the calculations. In case of distribution="dnorm", the closer to zero the intervals are, the better model performs. When distribution="dfnorm" or distribution="dchisq", the smaller, the better.

Function returns a list of a class "rmc", which contains the following variables:

  • meanMean values for each method.

  • intervalConfidence intervals for each method.

  • vlinesCoordinates used for style="l", marking the groups of methods.

  • groupsThe table containing the groups. TRUE - methods are in the same group, FALSE - they are not.

  • p.valuep-value for the test of the significance of the model. This is a log-likelihood ratios chi-squared test, comparing the model with the one with intercept only.

  • importanceThe weights of the estimated model in comparison with the model with the constant only. 0 means that the constant is better, 1 means that the estimated model is the best.

  • levelSignificance level.

  • modellm model produced for the calculation of the intervals.

  • styleStyle of the plot to produce.

  • selectThe selected variable to highlight.

Details

The test constructs the regression model of the kind:

y = b' X + e,

where y is the vector of the provided data (as.vector(data)), X is the matrix of dummy variables for each column of the data (forecasting method), b is the vector of coefficients for the dummies and e is the error term of the model.

Depending on the provided data, it might make sense to use different types of regressions. The function supports Gausian linear regression (distribution="dnorm", when the data is normal), advanced linear regression with folded normal distribution (distribution="dfnorm", for example, absolute errors, assuming that the original errors are normally distributed) and advanced linear regression with Chi-Squared distribution (distribution="dchisq", when the data is distributed as Chi^2, for example squared normal standard errors).

The advisable error measures to use in the test are RelMAE and RelMSE, which are unbiased and whose logarithms are symmetrically distributed (Davydenko & Fildes, 2013). In fact RelMSE should have F-distribution with h and h degrees of freedom and its logarithm is a log F distribution, because each MSE * h has chi-square(h) (assuming that the forecast error is normal).

As for RelMAE, its distribution is trickier, because each MAE has folded normal distribution (assuming that the original error is normal) and their ratio is something complicated, but tractable (Kim, 2006).

Still, given large samples, the parameters of the regression on logarithms of the both RelMAE and RelMSE should have normal distribution. Thus distribution="dnorm" can be used in this case (see examples).

The test is equivalent to nemenyi test, when applied to the ranks of the error measures on large samples.

There is also a plot() method that allows producing either "mcb" or "lines" style of plot. This can be regulated via plot(x, style="lines").

References

See Also

alm

Examples

Run this code
# NOT RUN {
N <- 50
M <- 4
ourData <- matrix(rnorm(N*M,mean=0,sd=1), N, M)
ourData[,2] <- ourData[,2]+1
ourData[,3] <- ourData[,3]+0.7
ourData[,4] <- ourData[,4]+0.5
colnames(ourData) <- c("Method A","Method B","Method C - long name","Method D")
rmc(ourData, distribution="dnorm", level=0.95)
# In case of AE-based measures, distribution="dfnorm" should be selected
rmc(abs(ourData), distribution="dfnorm", level=0.95)

# In case of SE-based measures, distribution="dchisq" should be selected
rmc(ourData^2, distribution="dchisq", level=0.95)

# APE-based measures should not be used in general...

# If RelMAE or RelMSE is used for measuring data, then it makes sense to use
# distribution="dnorm" and provide logarithms of the RelMAE, which can be approximated by
# normal distribution
ourData <- abs(ourData)
rmc(ourData / ourData[,1], distribution="dnorm", level=0.95)

# The following example should give similar results to nemenyi test on
# large samples, which compares medians of the distributions:
rmc(t(apply(ourData,1,rank)), distribution="dnorm", level=0.95)

# }

Run the code above in your browser using DataLab