data_cleansing: Data Cleaning

Description

The data_cleansing function is a simpler wrapper for data cleaning functions, such as delete variables that values are all NAs;checking dat and target format.;delete low variance variables.;replace null or NULL or blank with NA; encode variables which NAs & miss value rate is more than 95

Usage

data_cleansing(dat, target = NULL, x_list = NULL, obs_id = NULL,
  occur_time = NULL, pos_flag = NULL, miss_values = NULL,
  ex_cols = NULL, outlier_proc = TRUE, missing_proc = TRUE,
  low_var = TRUE, one_hot = FALSE, parallel = FALSE, note = FALSE,
  save_data = FALSE, file_name = NULL, dir_path = tempdir())

Arguments

dat

A data frame with x and target.

target

The name of target variable.

x_list

A list of x variables.

obs_id

The name of ID of observations.Default is NULL.

occur_time

The name of occur time of observations.Default is NULL.

pos_flag

The value of positive class of target variable, default: "1".

miss_values

Other extreme value might be used to represent missing values, e.g: -9999, -9998. These miss_values will be encoded to -1 or "Missing".

ex_cols

A list of excluded variables. Default is NULL.

outlier_proc

Logical, process outliers or not. Default is TRUE.

missing_proc

Logical, process nas or not. Default is TRUE.

low_var

Logical, delete low variance variables or not. Default is TRUE.

one_hot

Logical. If TRUE, one-hot_encoding of category variables. Default is FASLE.

parallel

Logical, parallel computing or not. Default is FALSE.

note

Logical. Outputs info. Default is TRUE.

save_data

Logical, save the result or not. Default is FALSE.

file_name

The name for periodically saved data file. Default is NULL.

dir_path

The path for periodically saved data file. Default is tempdir().

Value

A preprocessed data.frame

Examples

Run this code

# NOT RUN {
#data cleaning
dat_cl <- data_cleansing(dat = UCICreditCard[1:2000,],
                       target = "default.payment.next.month",
                       x_list = NULL,
                       obs_id = "ID",
                       occur_time = "apply_date",
                       ex_cols = c("PAY_6|BILL_"),
                       outlier_proc = TRUE,
                       missing_proc = TRUE,
                      one_hot = FALSE,
                       low_var = TRUE,
                       save_data = FALSE)

# }

Run the code above in your browser using DataLab

Last chance! 50% off unlimited learning