CBTF: Caught by the Fuzz! A minimalistic fuzz-test runner for R

The CBTF package implements a very simple mechanism for fuzz-testing functions in the public interface of an R package.

Fuzz testing helps identify functions that lack sufficient argument validation, and uncovers sets of inputs that, while valid by function signature, may cause issues within the function body.

The core functionality of the package is in the fuzz() function, which calls each provided function with a certain input and records the output produced. If an error or a warning is generated, this is captured and reported to the user, unless it matches a pattern of whitelisted messages, as specified in the ignore_patterns argument. The objects returned by fuzz() can be inspected with summary() and print().

Whitelisting can also be done after a fuzz run has been completed via the whitelist() function, so that only messages that need to be acted upon are actually shown. Using whitelist() has the advantage of not requiring to run the fuzzer over all functions and all inputs again.

Note that fuzz() uses the mirai package for asynchronous operations and parallelisation, and execution occurs on persistent background processes. These can be started automatically by specifying the daemons option; alternatively, they can be set up manually with the mirai::daemons() function; refer to the original mirai documentation for a complete description of its arguments and behaviour.

The helper function get_exported_functions() identifies the functions in the public interface of a given package, facilitating the generation of the list of functions to be fuzzed.

The helper function test_inputs() is invoked by fuzz() if the user doesn’t specify the set of inputs to be tested. By default it generates a large set of potentially problematic inputs, but these can be limited just to the desired classes of inputs.

Usage

This is a simple example that displays how to use CBTF to fuzz an R package. We consider mime because it is small enough to run quickly and is likely installed on most systems.

library(CBTF)
funs <- get_exported_functions("mime")
(res <- fuzz(funs, what = list(TRUE)))

## ℹ Fuzzing 2 functions with 1 input (using 2 daemons)
## ℹ 2 tests run  [4ms]
## ✖