datadr (version 0.8.4)

drQuantile: Sample Quantiles for 'ddf' Objects

Description

Compute sample quantiles for 'ddf' objects

Usage

drQuantile(x, var, by = NULL, probs = seq(0, 1, 0.005), preTransFn = NULL,
  varTransFn = identity, varRange = NULL, nBins = 10000, tails = 100,
  params = NULL, packages = NULL, control = NULL, ...)

Arguments

x
a 'ddf' object
var
the name of the variable to compute quantiles for
by
an optional variable name or vector of variable names by which to group quantile computations
probs
numeric vector of probabilities with values in [0-1]
preTransFn
a transformation function (if desired) to applied to each subset prior to computing quantiles (here it may be useful for adding a "by" variable that is not present) - note: this transformation should not modify var (use varTransFn
varTransFn
transformation to apply to variable prior to computing quantiles
varRange
range of x (can be left blank if summaries have been computed)
nBins
how many bins should the range of the variable be split into?
tails
how many exact values at each tail should be retained?
params
a named list of objects external to the input data that are needed in the distributed computing (most should be taken care of automatically such that this is rarely necessary to specify)
packages
a vector of R package names that contain functions used in fn (most should be taken care of automatically such that this is rarely necessary to specify)
control
parameters specifying how the backend should handle things (most-likely parameters to rhwatch in RHIPE) - see rhipeControl and localDiskContr
...
additional arguments

Value

  • data frame of quantiles q and their associated f-value fval. If by is specified, then also a variable group.

Details

This division-agnostic quantile calculation algorithm takes the range of the variable of interest and splits it into nBins bins, tabulates counts for those bins, and reconstructs a quantile approximation from them. nBins should not get too large, but larger nBins gives more accuracy. If tails is positive, the first and last tails ordered values are attached to the quantile estimate - this is useful for long-tailed distributions or distributions with outliers for which you would like more detail in the tails.

See Also

updateAttributes

Examples

Run this code
# break the iris data into k/v pairs
irisSplit <- list(
  list("1", iris[1:10,]), list("2", iris[11:110,]), list("3", iris[111:150,])
)
# represent it as ddf
irisSplit <- ddf(irisSplit, update = TRUE)

# approximate quantiles over the divided data set
probs <- seq(0, 1, 0.005)
iq <- drQuantile(irisSplit, var = "Sepal.Length", tails = 0, probs = probs)
plot(iq$fval, iq$q)

# compare to the all-data quantile "type 1" result
plot(probs, quantile(iris$Sepal.Length, probs = probs, type = 1))

Run the code above in your browser using DataCamp Workspace