datanugget (version 1.0.0)

create.DN: Create Data Nuggets

Description

This function draws a random sample of observations from a large dataset and creates data nuggets, a type of representative sample of the dataset, using a specified distance metric.

Usage

create.DN(x,
          RS.num = 2.5*(10^5),
          DN.num1 = 10^4,
          DN.num2 = 2000,
          dist.metric = "euclidean",
          seed = 291102,
          no.cores = (detectCores() - 1),
          make.pbs = TRUE)

Arguments

x

A data matrix (of class matrix, data.frame, or data.table) containing only entries of class numeric.

RS.num

The number of observations to sample from the data matrix. Must be of class numeric.

DN.num1

The number of initial data nugget centers to create. Must be of class numeric.

DN.num2

The number of data nuggets to create. Must be of class numeric.

dist.metric

The distance metric used to create the initial centers of data nuggets. Must be 'euclidean' or 'manhattan'.

seed

Random seed for replication. Must be of class numeric.

no.cores

Number of cores used for parallel processing. If '0' then parallel processing is not used. Must be of class numeric.

make.pbs

Print progress bars? Must be TRUE or FALSE.

Value

An object of class datanugget:

Data Nuggets

DN.num by (ncol(x)+3) data frame containing the information for the data nuggets created (index, center, weight, scale).

Data Nugget Assignments

Vector of length nrow(x) containing the data nugget assignment of each observation in x.

Details

Data nuggets are a representative sample meant to summarize Big Data by reducing a large dataset to a much smaller dataset by eliminating redundant points while also preserving the peripheries of the dataset. Each data nugget is defined by a center (location), weight (importance), and scale (internal variability). This function creates data nuggets using Algorithm 1 provided in the reference.

References

Data Nuggets: A Method for Reducing Big Data While Preserving Data Structure (Submitted for Publication, 2019)

Examples

# NOT RUN {
      ## small example
      X = cbind.data.frame(rnorm(10^4),
                           rnorm(10^4),
                           rnorm(10^4))

      suppressMessages({

        my.DN = create.DN(x = X,
                          RS.num = 10^3,
                          DN.num1 = 500,
                          DN.num2 = 250,
                          no.cores = 0,
                          make.pbs = FALSE)

      })

      my.DN$`Data Nuggets`
      my.DN$`Data Nugget Assignments`

    
# }
# NOT RUN {
      ## large example
      X = cbind.data.frame(rnorm(10^6),
                           rnorm(10^6),
                           rnorm(10^6),
                           rnorm(10^6),
                           rnorm(10^6))

      my.DN = create.DN(x = X,
                        RS.num = 10^5,
                        DN.num1 = 10^4,
                        DN.num2 = 2000)

      my.DN$`Data Nuggets`
      my.DN$`Data Nugget Assignments`

    
# }
# NOT RUN {
# }