The fuzzer calls each function in funs with the argument list provided in
args (each of its elements in turn modified by each object in what) and
records any errors or warnings that are thrown. If no error occurs within
the first timeout seconds, the execution of the function being fuzzed is
interrupted and the next one is started.
fuzz(
funs,
what = test_inputs(),
args = NULL,
package = NULL,
listify_what = FALSE,
ignore_patterns = "",
ignore_warnings = FALSE,
daemons = 2L,
timeout = 2
)An object of class cbtf that stores the results obtained for each of the
functions tested. This contains the following fields:
a list of data frames, each containing the results of fuzzing
all the functions in funs with one of the inputs in what. The
data frame contains the following columns and attributes:
- res: The result of the fuzz test, see below for the possible
values.
- msg: The error or warning message returned by the function, if
any.
- attr(*, "what"): The character representation of the input
tested.
a vector of names of the functions tested.
a named list of arguments, with names generated by deparsing
the args argument if not already provided.
a character string specifying the package name where
function names were searched, or NA if none was provided.
The value of the ignore_patterns argument.
The value of the ignore_warnings argument.
The res column in each of the data frames in the $runs field can
contain the following values:
OK: either no error or warning was produced (in which case, the msg
entry is left blank), or it was whitelisted (in which case, the message
received is stored in msg), or it was timed out (in which case, msg
records that a timeout was applied).
SKIP: no test was run, either because the given name cannot be found, or
it doesn't correspond to a function, or the function accepts no arguments,
or more arguments were provided than the function accepts;
the exact reason is given in msg.
WARN: a warning was thrown for which no whitelisting occurred and
ignore_warnings = FALSE; its message is stored in msg.
FAIL: an error was thrown for which no whitelisting occurred; its message
is stored in msg.
A character vector of function names to test. If a "package"
attribute is set and is no package argument is provided, functions
are loaded from the namespace specified in the attribute.
A list of objects; each is used, in turn, to modify the list of
arguments in args before calling each of the functions in funs.
If no inputs are provided, a default set of inputs generated by
test_inputs will be used. If set to NULL, then args must be
specified, and all functions will be called with that exact list of
arguments with no fuzzing occurring.
A list of default values for function arguments. Each argument
in the list is in turn replaced by each element of what, then each
modified argument list is used to fuzz the functions in funs. If
named arguments are present, their names are used as argument names
in the fuzzed functions. If NULL (default), only the first argument
of each function is fuzzed.
A character string specifying the name of the package to
search for functions. If NULL (default), the function will first
check the "package" attribute of funs, and if that is not set,
names will be searched in the global namespace.
Whether each input in what should also be tested
in its listified version (FALSE by default). When set to TRUE,
if what is list(x = x), the function will operate as if it
were list(x = x, "list(x)" = list(x)), for any input object x.
One or more strings containing regular expressions to match the errors to ignore. The strings "unused argument" and "is missing, with no default" are always ignored.
Whether warnings should be ignored (FALSE by
default).
Number of daemons to use (2 by default). As many mirai
daemons as specified will be started when entering the function and
closed at the end, unless active daemons are already available, in
which case the argument is ignored and the active daemons are used.
Number of seconds (2 by default) after which the function being fuzzed is interrupted with result status set to "OK".
An list of arguments to be passed to the functions being fuzzed can be
provided via the args argument. Each element in that list is modified in
turn by each object in what and the resulting list of arguments is then
passed to each function via do.call(). If more arguments are given than
the number formal arguments accepted by a function, that function will
produce a "SKIP" result.
If arguments are named, they will be passed with their name to the fuzzed
functions. If a function doesn't have a formal argument of that name and
doesn't accept ..., then the standard R behaviour is to return an "unused
argument" error. This is whitelisted by default in fuzz(), and the
corresponding result status is set to "OK".
It is possible to define arguments that should remain unchanged while
fuzzing by prefixing their name with ... These arguments will use the
values assigned in args without modification. For example, to ensure that
the argument na.rm is always set to TRUE, it should be specified as
..na.rm = TRUE in args. If all elements in args are fixed, what is
ignored and all functions in funs will be called with the provided args
list.
The implementation uses mirai as a backend to execute tasks asynchronously
in parallel worker processes. The function can start a pool of persistent
background processes (daemons) of size given by the daemons argument
(note that starting more daemons than available cores yields no benefit).
Alternatively, the function can also make use of already active daemons
started with the mirai::daemons function: this allows to control in
greater detail the number of processes to use, which can also be remote.
In order to reduce the number of false positive results produced, this function applies the following set rules, to establish if an error or warning condition should ignored (whitelisting):
If the name of the function appears in the error or warning message, as it is considered that the condition has been handled by the developer.
If the error or warning message contains the text "is missing, with no default", which is produced when a missing argument is used without a value being assigned to it.
If the error or warning message contains any of the patterns specified
in ignore_patterns.
If a warning is thrown but ignore_warnings = TRUE is set.
In all whitelisted cases, the result is "OK", and the message that
was received is stored in the $msg field (see the Value section).
Note: Whitelisting can also be applied post-hoc on the results of a fuzz run using the whitelist function.
get_exported_functions, test_inputs, whitelist, summary.cbtf, print.cbtf
## set up persistent background processes
mirai::daemons(2L)
## this should produce no errors
res <- fuzz(funs = c("list", "matrix", "mean"),
what = test_inputs(c("numeric", "raw")))
summary(res)
## display all results even for successful tests
print(res, show_all = TRUE)
## this will catch an error (false positive)
fuzz(funs = "matrix", what = test_inputs("scalar"))
## apply a whitelist pattern to remove the false positive
fuzz(funs = "matrix", what = test_inputs("scalar"),
ignore_patterns = "'data' must be of a vector type")
## close the background processes
mirai::daemons(0L)
Run the code above in your browser using DataLab