Visual Regression Testing and Graphical Diffing
vdiffr is an extension to testthat that makes it easy to
add tests for graphics. It provides a Shiny application to manage
the test cases.
vdiffr is an extension to the package testthat that makes it easy to test for visual regressions. It provides a Shiny app to manage failed tests and visually compare a graphic to its expected output.
Get the development version from github with:
# install.packages("devtools") devtools::install_github("lionel-/vdiffr")
How to use vdiffr
vdiffr integrates with testthat through the
expectation. It takes as arguments:
A title. This title is used in two ways. First, the title is standardised (it is converted to lowercase and any character that is not alphanumeric or a space is turned into a dash) and used as filename for storing the figure. Secondly, with ggplot2 figures the title is automatically added to the plot with
ggtitle()(only if no ggtitle has been set).
A figure. This can be a ggplot object, a recordedplot, a function to be called, or more generally any object with a
Optionally, a path where to store the figures, relative to
tests/figs/. They are stored in a subfolder according to the current testthat context by default. Supply
pathto change the subfolder.
For example, the following tests will create figures in
context("Histograms") disp_hist_base <- function() hist(mtcars$disp) disp_hist_ggplot <- ggplot(mtcars, aes(disp)) + geom_histogram() vdiffr::expect_doppelganger("Base graphics histogram", disp_hist_base) vdiffr::expect_doppelganger("ggplot2 histogram", disp_hist_ggplot)
Note that in addition to automatic ggtitles, ggplot2 figures are
assigned the minimalistic theme
theme_test() (unless they already
have been assigned a theme).
You can run the tests the usual way, for example with
devtools::test(). New cases for which you just wrote an expectation
will be skipped. Failed tests will show as an error.
Managing the tests
When you have added new test cases or detected regressions, you can
manage those from the R command line with the functions
However it's easier to run the shiny application
With this app you can:
Check how a failed case differs from its expected output using three widgets: Toggle (click to swap the images), Slide and Diff. If you use Github, you may be familiar with the last two.
Validate cases. You can do so groupwise (all new cases or all failed cases) or on a case by case basis. When you validate a failed case, the old expected output is replaced by the new one.
Delete orphaned cases. During a refactoring of your unit tests, some visual expectations may be removed or renamed. This means that some unused figures will linger in the
tests/figs/folder. These figures appear in the Shiny application under the category "Orphaned" and can be cleaned up from there.
package as first
argument, the path to your package sources. This argument has exactly
the same semantics as in devtools. You can use vdiffr tools the same
way as you would use
devtools::check(), for example. The default is
".", meaning that the package is expected to be found in the current
All validated cases are stored in
tests/figs/. This folder may be
handy to showcase the different graphs offered in your package. You
can also keep track of how your plots change as you tweak their layout
and add features by checking the history on Github.
An addin to launch
manage_cases() is provided with vdiffr. Use the
addin menu to launch the Shiny app in an RStudio dialog.
To use the Shiny app as part of ESS devtools integration with
C-v, include something like this in your init file:
(defun ess-r-vdiffr-manage-cases () (interactive) (ess-r-package-send-process "vdiffr::manage_cases(%s)\n" "Manage vdiffr cases for %s")) (define-key ess-r-package-dev-map "\C-v" 'ess-r-vdiffr-manage-cases)
Continuous integration on Travis
To work properly, vdiffr requires the C library
2.6.1 or later. The FreeType libraries available by default on Travis'
Linux platforms are not this recent yet. Some adjustments to the
travis.yml file are thusrequired.
Ubuntu Precise (the default):
addons: apt: sources: - debian-sid packages: - libfreetype6
sudo: required before_install: [ "sudo add-apt-repository \"deb http://archive.ubuntu.com/ubuntu/ xenial main\" -y", "sudo apt-get update -q", "sudo apt-get install libfreetype6" ]
macOS with XCode 6.1:
osx_image: beta-xcode6.1 disable_homebrew: true latex: false
macOS with XCode 7.2:
osx_image: xcode7.2 brew_packages: cairo latex: false
vdiffr currently uses svglite to save the plots in a text format that
makes it easy to perform comparisons. This makes the test cases
dependent on that package as the SVG translation of the plot may
change across different versions of svglite (though that should not
happen often). For this reason, whenever you validate a graphical test
tests/figs/deps.txt file is updated with a note containing
the svglite version. This works the same way as the roxygen version
Your graphics might be dependent on other packages besides svglite. If
your package is an extension to ggplot2 for instance, the appearance
of your plot may change as ggplot2 evolves (as with the 2.0 version
which tweaked the grayness of the background color among other
changes). For this reason,
expect_doppelganger() adds a dependence
on ggplot2 when you supply a ggplot2 object. You can also manually add
a dependency on any other package by calling
anywhere in a test file.
vdiffr extends testthat through a custom
are classes (R6 classes in recent versions of testthat) whose
instances collect cases and output a summary of the tests. While
reporters are usually meant to provide output for the end user, you
can also use them in functions to interact with testthat.
vdiffr has a
that does nothing but activate a collecter for the visual test
devtools::test() with this
expect_doppelganger() is called, it first checks
whether the case is new or failed. If that's the case, and if it finds
that vdiffr's collecter is active, it calls the collecter, which in
turns records the current test case.
This enables the user to run the tests with the usual development tools and get feedback in the form of skipped or failed cases. On the other hand, when vdiffr's tools are called, we collect information about the tests of interest and wrap them in a data structure.
Comparing SVG files is convenient and should work correctly in most situations. However, SVG is not suitable for tracking really subtle changes and regressions. See vdiffr's issue #1 for a discussion on this. vdiffr may gain additional comparison backends in the future to make the tests more stringent.
Functions in vdiffr
|expect_doppelganger||Does a figure look like its expected output?|
|add_dependency||Add a vdiffr dependency|
|vdiffrAddin||RStudio Addin for managing visual cases|
|collect_cases||Collect and validate cases|
|htmlwidgets||HTML Widgets for graphical comparison|
|shinybindings||Shiny bindings for graphical comparison widgets|
|manage_cases||Manage visual test cases with a Shiny app|
Last month downloads
|Remotes||lionel-/fontquiver, hadley/svglite, davidgohel/gdtools#32|
Include our badge in your README