sparklyr.flint (version 0.2.2)

summarize_weighted_covar: Weighted covariance summarizer

Description

Compute unbiased weighted covariance between values from `xcolumn` and `ycolumn` within each time window or within each group of records with identical timestamps, using values from `weight_column` as relative weights, and store results in a new column named `<xcolumn>_<ycolumn>_<weight_column>_weightedCovariance`

Usage

summarize_weighted_covar(
  ts_rdd,
  xcolumn,
  ycolumn,
  weight_column,
  window = NULL,
  key_columns = list()
)

Arguments

ts_rdd

Timeseries RDD being summarized

xcolumn

Column representing the first random variable

ycolumn

Column representing the second random variable

weight_column

Column specifying relative weight of each data point

window

Either an R expression specifying time windows to be summarized (e.g., `in_past("1h")` to summarize data from looking behind 1 hour at each time point, `in_future("5s")` to summarize data from looking forward 5 seconds at each time point), or `NULL` to compute aggregate statistics on records grouped by timestamps

key_columns

Optional list of columns that will form an equivalence relation associating each record with the time series it belongs to (i.e., any 2 records having equal values in those columns will be associated with the same time series, and any 2 records having differing values in those columns are considered to be from 2 separate time series and will therefore be summarized separately) By default, `key_colums` is empty and all records are considered to be part of a single time series.

Value

A TimeSeriesRDD containing the summarized result

See Also

Other summarizers: ols_regression(), summarize_avg(), summarize_corr2(), summarize_corr(), summarize_count(), summarize_covar(), summarize_dot_product(), summarize_ema_half_life(), summarize_ewma(), summarize_geometric_mean(), summarize_kurtosis(), summarize_max(), summarize_min(), summarize_nth_central_moment(), summarize_nth_moment(), summarize_product(), summarize_quantile(), summarize_skewness(), summarize_stddev(), summarize_sum(), summarize_var(), summarize_weighted_avg(), summarize_weighted_corr(), summarize_z_score()

Examples

# NOT RUN {
library(sparklyr)
library(sparklyr.flint)

sc <- try_spark_connect(master = "local")

if (!is.null(sc)) {
  sdf <- copy_to(sc, tibble::tibble(t = seq(10), u = rnorm(10), v = rnorm(10), w = 1.1^seq(10)))
  ts <- fromSDF(sdf, is_sorted = TRUE, time_unit = "SECONDS", time_column = "t")
  ts_weighted_covar <- summarize_weighted_covar(
    ts,
    xcolumn = "u", ycolumn = "v", weight_column = "w", window = in_past("3s")
  )
} else {
  message("Unable to establish a Spark connection!")
}

# }