Learn R Programming
greybox (version 0.5.3)

rmc: Regression for Multiple Comparison

Description

RMC stands for "Regression for Multiple Comparison", referring to the comparison of forecasting methods. This is a parametric test for the comparison of means of several distributions This test is a parametric counterpart of Nemenyi / MCB test (Demsar, 2006) and uses asymptotic properties of regression models. It relies on distributional assumptions about the provided data. For instance, if the mean forecast errors are used, then it is safe to assume that the regression model constructed on them will have symmetrically distributed residuals, thus normal regression can be used for the parameters estimation.

Usage

rmc(data, distribution = c("dlnorm", "dnorm", "dfnorm"), level = 0.95,
  outplot = c("mcb", "lines", "none"), select = NULL, ...)

Arguments

data

Matrix or data frame with observations in rows and variables in columns.

distribution

Type of the distribution to use. This value is passed to alm() function. "dlnorm" would lead to the alm with log normal distribution. If this is a clear forecast error, then "dnorm" might be appropriate, leading to a simple Gausian linear regression. "dfnorm" would lead to an alm model with folded normal distribution. You can try some other distributions, but don't expect anything meaningful.

level

The width of the confidence interval. Default is 0.95.

outplot

What type of plot to use after the calculations. This can be either "MCB" ("mcb"), or "Vertical lines" ("lines"), or nothing ("none"). You can also use plot method on the produced object in order to get the same effect.

select

What column of data to highlight on the plot. If NULL, then the method with the lowest value is selected.

...

Other parameters passed to plot function

Value

If outplot!="none", then the function plots the results after all the calculations. In case of distribution="dnorm", the closer to zero the intervals are, the better model performs. When distribution="dlnorm" or distribution="dfnorm", the lower, the better.

Function returns a list of a class "rmc", which contains the following variables:

  • meanMean values for each method.

  • intervalConfidence intervals for each method.

  • vlinesCoordinates used for outplot="l", marking the groups of methods.

  • groupsThe table containing the groups. TRUE - methods are in the same group, FALSE - they are not.

  • methodsSimilar to group parameter, but with a slightly different presentation.

  • p.valuep-value for the test of the significance of the model. This is a log-likelihood ratios chi-squared test, comparing the model with the one with intercept only.

  • importanceThe weights of the estimated model in comparison with the model with the constant only. 0 means that the constant is better, 1 means that the estimated model is the best.

  • levelSignificance level.

  • modellm model produced for the calculation of the intervals.

  • outplotStyle of the plot to produce.

  • selectThe selected variable to highlight.

Details

The test constructs the regression model of the type:

y = b' X + e,

where y is the vector of the provided data (as.vector(data)), X is the matrix of dummy variables for each column of the data (forecasting method), b is the vector of coefficients for the dummies and e is the error term of the model.

Depending on the provided data, it might make sense to use different types of regressions. The default one is the log normal distribution for the relative error measures. The type of distribution is regulated with distribution and is restricted by the values of it from the alm function.

The advisable error measures to use in the test are relative measures, such as RelMAE, RelRMSE, RelAME. They are unbiased and their logarithms are symmetrically distributed (Davydenko & Fildes, 2013). Although their distributions are not log normal, given the typically large samples of datasets, the Central Limit Theorem helps in the adequate construction of the confidence intervals for the parameters.

The test is equivalent to Nemenyi test, when applied to the ranks of the error measures on large samples with distribution="dnorm".

There is also a plot() method that allows producing either "mcb" or "lines" style of plot. This can be regulated via plot(x, outplot="lines").

References

See Also

alm

Examples

Run this code
# NOT RUN {
N <- 50
M <- 4
ourData <- matrix(rnorm(N*M,mean=0,sd=1), N, M)
ourData[,2] <- ourData[,2]+4
ourData[,3] <- ourData[,3]+3
ourData[,4] <- ourData[,4]+2
colnames(ourData) <- c("Method A","Method B","Method C - long name","Method D")
rmc(ourData, distribution="dnorm", level=0.95)
# In case of AE-based measures, distribution="dfnorm" should be selected
rmc(abs(ourData), distribution="dfnorm", level=0.95)

# APE-based measures should not be used in general...

# If RelMAE or RelMSE is used for measuring data, then it makes sense to use
# distribution="dlnorm" for the RelMAE / RelMSE, as it can be approximated by
# log normal distribution, because according to Davydenko & Fildes (2013) the
# logarithms of these measures have symmetric distribution.
ourTest <- rmc((abs(ourData) / rfnorm(N, 0.3, 1)), distribution="dlnorm", level=0.95)
# The exponents of mean values from this function will correspond to the
# geometric means of RelMAE / RelMSE.
exp(ourTest$mean)
# The same is for the intervals:
exp(ourTest$interval)

# You can also reproduce plots in different styles:
plot(ourTest, outplot="lines")

# Or you can use the default "mcb" style and set additional parameters for the plot():
par(mar=c(2,2,4,0)+0.1)
plot(ourTest, main="Four methods")

# The following example should give similar results to Nemenyi test on
# large samples, which compares medians of the distributions:
rmc(t(apply(ourData,1,rank)), distribution="dnorm", level=0.95)

# You can also give a try to SE-based measures with distribution="dchisq":
rmc(ourData^2, distribution="dchisq", level=0.95)

# }

Run the code above in your browser using DataLab