The rows_distinct() validation function, the expect_rows_distinct()
expectation function, and the test_rows_distinct() test function all check
whether row values (optionally constrained to a selection of specified
columns) are, when taken as a complete unit, distinct from all other units
in the table. The validation function can be used directly on a data table or
with an agent object (technically, a ptblank_agent object) whereas the
expectation and test functions can only be used with a data table. The types
of data tables that can be used include data frames, tibbles, database tables
(tbl_dbi), and Spark DataFrames (tbl_spark). As a validation step or as
an expectation, this will operate over the number of test units that is equal
to the number of rows in the table (after any preconditions have been
applied).
rows_distinct(
x,
columns = NULL,
preconditions = NULL,
actions = NULL,
step_id = NULL,
label = NULL,
brief = NULL,
active = TRUE
)expect_rows_distinct(
object,
columns = NULL,
preconditions = NULL,
threshold = 1
)
test_rows_distinct(object, columns = NULL, preconditions = NULL, threshold = 1)
A data frame, tibble (tbl_df or tbl_dbi), Spark DataFrame
(tbl_spark), or, an agent object of class ptblank_agent that is created
with create_agent().
The column (or a set of columns, provided as a character vector) to which this validation should be applied.
expressions used for mutating the input table before
proceeding with the validation. This is ideally as a one-sided R formula
using a leading ~. In the formula representation, the . serves as the
input data table to be transformed (e.g.,
~ . %>% dplyr::mutate(col = col + 10).
A list containing threshold levels so that the validation step
can react accordingly when exceeding the set levels. This is to be created
with the action_levels() helper function.
One or more optional identifiers for the single or multiple
validation steps generated from calling a validation function. The use of
step IDs serves to distinguish validation steps from each other and provide
an opportunity for supplying a more meaningful label compared to the step
index. By default this is NULL, and pointblank will automatically
generate the step ID value (based on the step index) in this case. One or
more values can be provided, and the exact number of ID values should (1)
match the number of validation steps that the validation function call will
produce (influenced by the number of columns provided), (2) be an ID
string not used in any previous validation step, and (3) be a vector with
unique values.
An optional label for the validation step.
An optional, text-based description for the validation step.
A logical value indicating whether the validation step should
be active. If the step function is working with an agent, FALSE will make
the validation step inactive (still reporting its presence and keeping
indexes for the steps unchanged). If the step function will be operating
directly on data, then any step with active = FALSE will simply pass the
data through with no validation whatsoever. The default for this is TRUE.
A data frame, tibble (tbl_df or tbl_dbi), or Spark
DataFrame (tbl_spark) that serves as the target table for the expectation
function or the test function.
A simple failure threshold value for use with the
expectation function. By default, this is set to 1 meaning that any
single unit of failure in data validation results in an overall test
failure. Whole numbers beyond 1 indicate that any failing units up to
that absolute threshold value will result in a succeeding testthat
test. Likewise, fractional values (between 0 and 1) act as a
proportional failure threshold.
For the validation function, the return value is either a
ptblank_agent object or a table object (depending on whether an agent
object or a table was passed to x). The expectation function invisibly
returns its input but, in the context of testing data, the function is
called primarily for its potential side-effects (e.g., signaling failure).
The test function returns a logical value.
2-15
We can specify the constraining column names in quotes, in vars(), and with
the following tidyselect helper functions: starts_with(),
ends_with(), contains(), matches(), and everything().
Having table preconditions means pointblank will mutate the table just
before interrogation. Such a table mutation is isolated in scope to the
validation step(s) produced by the validation function call. Using
dplyr code is suggested here since the statements can be translated to
SQL if necessary. The code is most easily supplied as a one-sided R
formula (using a leading ~). In the formula representation, the . serves
as the input data table to be transformed (e.g.,
~ . %>% dplyr::mutate(col_a = col_b + 10)). Alternatively, a function could
instead be supplied (e.g.,
function(x) dplyr::mutate(x, col_a = col_b + 10)).
Often, we will want to specify actions for the validation. This argument,
present in every validation function, takes a specially-crafted list
object that is best produced by the action_levels() function. Read that
function's documentation for the lowdown on how to create reactions to
above-threshold failure levels in validation. The basic gist is that you'll
want at least a single threshold level (specified as either the fraction of
test units failed, or, an absolute value), often using the warn_at
argument. This is especially true when x is a table object because,
otherwise, nothing happens. For the col_vals_*()-type functions, using
action_levels(warn_at = 0.25) or action_levels(stop_at = 0.25) are good
choices depending on the situation (the first produces a warning when a
quarter of the total test units fails, the other stop()s at the same
threshold level).
Want to describe this validation step in some detail? Keep in mind that this
is only useful if x is an agent. If that's the case, brief the agent
with some text that fits. Don't worry if you don't want to do it. The
autobrief protocol is kicked in when brief = NULL and a simple brief will
then be automatically generated.
Other validation functions:
col_exists(),
col_is_character(),
col_is_date(),
col_is_factor(),
col_is_integer(),
col_is_logical(),
col_is_numeric(),
col_is_posix(),
col_schema_match(),
col_vals_between(),
col_vals_equal(),
col_vals_expr(),
col_vals_gte(),
col_vals_gt(),
col_vals_in_set(),
col_vals_lte(),
col_vals_lt(),
col_vals_not_between(),
col_vals_not_equal(),
col_vals_not_in_set(),
col_vals_not_null(),
col_vals_null(),
col_vals_regex(),
conjointly()
# NOT RUN {
# Create a simple table with three
# columns of numerical values
tbl <-
dplyr::tibble(
a = c(5, 7, 6, 5, 8, 7),
b = c(7, 1, 0, 0, 8, 3),
c = c(1, 1, 1, 3, 3, 3)
)
# Validate that when considering only
# data in columns `a` and `b`, there
# are no duplicate rows (i.e., all
# rows are distinct)
agent <-
create_agent(tbl = tbl) %>%
rows_distinct(vars(a, b)) %>%
interrogate()
# Determine if these column
# validations have all passed
# by using `all_passed()`
all_passed(agent)
# }
Run the code above in your browser using DataLab