fuzz: Fuzz-test the specified functions

Description

This function calls each of the functions in funs with each of the objects specified in what, recording if any errors or warnings are thrown in the process.

Usage

fuzz(
  funs,
  what = test_inputs(),
  package = NULL,
  listify_what = FALSE,
  ignore_patterns = "",
  ignore_warnings = FALSE
)

Value

An object of class cbtf that stores the results obtained for each of the functions tested. This contains the following fields:

runs: a list of data frames, each containing the results of fuzzing all the functions in funs with one of the inputs in what. The data frame contains the following columns and attributes:
- res: The result of the fuzz test, see below for the possible values.
- msg: The error or warning message returned by the function, if any.
- attr(*, "what"): The character representation of the input tested.
funs: a vector of names of the functions tested.
package: a character string specifying the package name where function names were searched, or NA if none was provided.
ignore_patterns: The value of the ignore_patterns argument.
ignore_warnings: The value of the ignore_warnings argument.

The res column in each of the data frames in the $runs field can contain the following values:

OK: either no error or warning was produced (in which case, the msg entry is left blank), or it was whitelisted (in which case, the message received is stored in msg).
SKIP: no test was run, either because the given name cannot be found, or it doesn't correspond to a function, or the function accepts no arguments, or the function contains a call to readline; the exact reason is given in msg.
WARN: a warning was thrown for which no whitelisting occurred and ignore_warnings = FALSE; its message is stored in msg.
FAIL: an error was thrown for which no whitelisting occurred; its message is stored in msg.

Arguments

funs: A character vector of function names to test. If a "package" attribute is set and is no package argument is provided, functions are loaded from the namespace specified in the attribute.
what: A list of objects to be passed, one at a time, as the first argument to each function in funs. Ideally, the list should be named, so that each input can be pretty-printed with its corresponding name; function namify provides an automatic way to create a named list. For unnamed lists, a deparsed representation of the inputs will be used, which may appear unwieldy in some cases. If no inputs are provided, a default set of inputs generated by test_inputs will be used.
package: A character string specifying the name of the package to search for functions. If NULL (default), the function will first check the "package" attribute of funs, and if that is not set, names will be searched in the global namespace.
listify_what: Whether each input in what should also be tested in its listified version (FALSE by default). When set to TRUE, if what is list(x = x), the function will operate as if it were list(x = x, "list(x)" = list(x)), for any input object x.
ignore_patterns: One or more strings containing regular expressions to match the errors to ignore. The string "is missing, with no default" is always ignored.
ignore_warnings: Whether warnings should be ignored (FALSE by default).

Details

In order to reduce the number of false positive results produced, this function applies the following set rules, to establish if an error or warning condition should ignored (whitelisting):

If the name of the function appears in the error or warning message, as it is considered that the condition has been handled by the developer.
If the error or warning message contains the text "is missing, with no default", which is produced when a missing argument is used without a value being assigned to it.
If the error or warning message contains any of the patterns specified in ignore_patterns.
If a warning is thrown but ignore_warnings = TRUE is set.

In all whitelisted cases, the result is "OK", and the message that was received is stored in the $msg field (see the Value section).

Examples

Run this code

## this should produce no errors
res <- fuzz(funs = c("list", "matrix", "mean"),
            what = test_inputs(c("numeric", "raw")))
summary(res)

## display all results even for successful tests
print(res, show_all = TRUE)

## this will catch an error (false positive)
fuzz(funs = "matrix",  what = test_inputs("scalar"))