In order to check the correct behavior of an exercise it is compiled several times. In each iteration the objects created by the exercise are collected and its values can be inspected.
stresstest_exercise(file, n = 100, verbose = TRUE, seeds = NULL,
stop_on_error = length(as.character(unlist(file))) < 2, timeout = NULL,
maxit = getOption("num_to_schoice_maxit", -10000L), ...)# S3 method for stress
plot(x, type = c("overview", "solution", "rank", "runtime", "warnings", "error"),
threshold = NULL, variables = NULL, spar = TRUE, ask = TRUE, ...)
# S3 method for stress
summary(object, ...)
Function stresstest_exercise returns an object of class "stress" (a named list) with
the following elements:
The seeds that where used.
Compiling times for each iteration.
A data frame of length 1 objects that are created by the exercise.
The numeric solution, availability is depending on the type of exercise.
A matrix indicating the position of correct solutions.
The rank of the correct solution, only available for choice exercises.
The number of correct answers in multiple choice type questions.
If stop_on_error = FALSE the function also records warnings and errors
thrown. Depending on whether or not warnings or errors have been recorded, the
following elements will be added to the object returned.
Named vector with total number of warnings and errors recorded. There can be more warnings than randomizations, if exercises throw multiple warnings.
Only added if at least one warning was recorded, character vector.
Warning messages and number of occurrences, or NA if no warnings were raised.
Only added if at least one error was recorded, character vector.
Error message or NA if no error was thrown.
character. A specification of an exercise file. If multiple files
should be tested, argument file can also be a vector, matrix or list of
files. The latter case sets argument plot = FALSE.
integer. The number of replications.
logical. Should the seeds used for compiling the exercise be prompted on the console.
The seeds that should be used when compiling the exercise. The
default is seeds = 1:n.
logical. Should the function stop on error or return the seed, the file name and the error message. Useful when testing a number of exercises.
NULL or numeric vector. See section 'Timeout' for details.
numeric. Maximum number of iterations passed on to num_to_schoice,
in case that is used for finding a suitable set of wrong solutions for a question list.
Exceeding abs(maxit) yields either a warning (if maxit > 0) or an
error (if maxit < 0).
An object returned from function stresstest_exercise.
character. type == "overview" plots the basic overview, i.e, the runtimes,
numeric solutions, position/number/rank of correct solution(s), if available.
If type == "solution", the numeric solutions
are plotted against all input parameters stored in the objects element of x.
type == "rank" draws spineplots of ranks
vs. input parameters stored in objects. type == "runtime" plots the compiling
runtimes vs. objects. Type "warning" and "error" plots how errors
have been logged (if any) and how often and where warnings were thrown.
numeric. Can be used to set a threshold, e.g., for numeric solutions a factor
is created, factor(solution <= threshold), that is used on the y-axis of a
spineplot.
character. The variables that should be used from the objects for
plotting.
logical. Should graphical parameters be set or not.
logical. For multiple plots, should the user be asked to hit the return key to see the next plot.
Arguments passed to xexams.
Stresstest allows to set execution timeouts. By default, timeout is set NULL
(no limits). If timeout is a numeric vector of positive values:
If timeout is of length 1: Used to limit execution time (cpu and elapsed time)
per randomization to timeout seconds.
If timeout is of length 2: Specifies cpu time (timeout[1L]) and
per randomization time elapsed (timeout[2L]).
If timeout is of length 3: The first two elements are used to specify
cpu/time elapsed per randomization (see above), the thirs element specifies the
overall time elapsed for all randomizations.
The thirs element (timeout[3]) is checked after each iteration. If a question
stalls (infinite loop) this can still cause stresstest() to stall as well.
In order to check the correct behavior of an exercise function stresstest_exercise runs
xexams n times using different seeds. If an error occurs when compiling,
the error can be reproduced by setting the seed that is prompted at the console and create
the exercise again, e.g., with exams2html. This way errors can be detected
systematically.
All objects with length 1, which are created by the exercise, are collected in a data frame. These objects are assumed to be input parameters that control the output of the exercise. This can be used to detect certain input values that, e.g., lead to very long run times, or drive the number of correct answers in multiple choice exercises, etc.
For single and multiple choice type question the position(s) of the correct solution(s) is
returned. For single choice questions that are created from a numeric version, e.g., using
function num_to_schoice the answers are again converted to numeric and the rank
of the correct solution is reported. The rank is sometimes heavily driven by some input
parameters, e.g., the correct solution is always the largest or the smallest.
For non-numeric choice questions, the rank is based on the lexicographical order of
the answerlist.
xexams, num_to_schoice
if (FALSE) ## Stress testing.
t1 <- stresstest_exercise("tstat.Rmd", n = 100)
t2 <- stresstest_exercise("tstat2.Rmd", n = 100)
## Plotting.
plot(t1, type = "overview")
plot(t1, type = "solution")
plot(t1, type = "solution", threshold = 30)
plot(t2, type = "rank")
plot(t2, type = "runtime")
## For custom inspection, object can be
## transformed to a data.frame.
head(as.data.frame(t2))
## Multiple testing.
files <- list(
"boxplots.Rmd",
c("tstat.Rmd", "ttest.Rmd", "confint.Rmd"),
c("regression.Rmd", "anova.Rmd"),
"scatterplot.Rmd",
"relfreq.Rmd"
)
t3 <- stresstest_exercise(files, n = 100)
plot(t3)
Run the code above in your browser using DataLab