dataprep

Fast, efficient, and versatile data preprocessing and reshaping tools for R, with C++ / OpenMP / SIMD backends.

In one paragraph

dataprep provides an opinionated, high-performance pipeline for cleaning tabular and time-series data. The 0.1.8 release rewrites the cleaning routines in C++ and delivers a speedup over 0.1.5 that ranges from about 1.0× (for varidele on some full-year data) to about 1146× (for condextr on full-year Ubuntu data). The melt() and dcast() reshaping functions are benchmarked against every one of the seven major alternatives in the R and Python ecosystems, at every tested scale (from 1,000 to 100,000,000 rows), on two reference hosts; the resulting speed-up ranges are 0.6–1628.9× for melt() and 1.9–799.8× for dcast(). On both hosts the output is identical to reshape2, data.table, tidyr, pandas, polars, dask, and duckdb, within tol = 1e-12.

Why dataprep

dataprep provides a coherent, opinionated pipeline for preprocessing tabular and time-series data:

• Variable deletion by missing-value fraction (varidele).
• Observation deletion by consecutive missing runs (obsedele).
• Outlier removal by point-by-point weighted conditional extremum (condextr) or by percentile (percoutl).
• Missing-value imputation within short periods (shorvalu) or by linear / LOCF / NOCB / mean / median (impute_missing).
• Fast reshaping between wide and long formats (melt, dcast) with SIMD + OpenMP.
• Descriptive statistics, diagnostics, transformation, standardization, encoding, validation, and reporting.
• Time-series tools: detrending, diurnal-cycle removal, rolling statistics, lags, resampling, decomposition, drift detection, day/night and season flags.
• Fit / transform interfaces (prep_fit, prep_transform) that prevent data leakage during preprocessing.

Most heavy routines are written in C++ with Rcpp. Since 0.1.8, many operations are parallelized with OpenMP and vectorized with AVX2 / AVX-512 when the hardware supports it.

Design philosophy

The cleaning pipeline is organised around four sequential steps, each addressing a distinct failure mode of high-resolution environmental data:

1. Variable deletion. Drop size bins whose missing fraction exceeds a threshold, so downstream interpolation never has to extrapolate from far-away anchors.

2. Observation deletion. Drop rows whose selected columns contain a consecutive missing run longer than half minutes on both sides. Every remaining point then has a trustworthy anchor within half minutes.

3. Conditional extremum outlier removal. A single value can be a global maximum and still be legitimate, or vice versa. condextr() judges each candidate in context.

4. Short-period grouping interpolation. After steps 1–3, remaining NAs sit inside short gaps with a valid anchor within half minutes. shorvalu() interpolates within each short segment only.

   Interpolating across a long gap silently mixes two physically distinct regimes and can create new outliers at the segment boundary. Grouping by short segments keeps the interpolation local.

Steps 1–4 are wrapped by dataprep() for one-call use. The design reasoning is documented in full in vignette("dataprep-philosophy"). data1 in this package is the already-aggregated seven-column version of the same dataset; it is not a useful input for the cleaning pipeline.

Installation

Recommended (also builds the vignettes locally; needs pandoc and the R packages knitr and rmarkdown):

# install.packages("remotes")
remotes::install_github("chunshengliang/dataprep", build_vignettes = TRUE)

Fallback (no extra dependencies):

remotes::install_github("chunshengliang/dataprep")

The package requires a C++17 compiler (Rtools on Windows, Xcode / clang on macOS, gcc on Linux). The build_vignettes = TRUE variant additionally needs pandoc and the R packages knitr and rmarkdown; if any of those is missing, remotes will fail. Vignettes are also available on the package website: https://chunshengliang.github.io/dataprep/articles/.

Note for Windows users

When installing from GitHub with remotes::install_github(), Windows users may see:

Warning: file 'dataprep/configure' did not have execute permissions: corrected

Warning: file 'dataprep/cleanup' did not have execute permissions: corrected

This is expected and harmless. Windows NTFS does not preserve Unix execute bits, so R CMD build corrects them automatically. The configure.win and cleanup.win scripts still run, and the package installs and works normally — the warning does not affect any functionality in any way. Linux, macOS, and CRAN checks do not emit this warning, and Windows users installing the CRAN binary package with install.packages("dataprep") are not affected either.

Quick start

library(dataprep)

# The size-bin columns are the ones whose names are numeric
# (1.00, 1.12, ..., 1000). The four non-size columns
# (`date`, `tconc`, `TPNC`, `monthyear`) are excluded by this
# pattern.
size_bins <- grep("^[-+]?[0-9]*\\.?[0-9]+$", names(data))

cleaned <- dataprep(
  data,
  cols       = size_bins,
  group      = 4,        # monthyear
  interval   = 10,
  times      = 10,
  intervals  = 30
)
dim(cleaned)

Performance

melt() and dcast() are benchmarked against all 7 major alternatives across 10 shapes and 6 scales (1,000 to 100,000,000 rows). Every cell is measured with a C++ steady-clock timer and an adaptive times rule (20 / 15 / 10 / 5 / 1 iterations based on warmup time). Two reference hosts were used.

Reference host A — Ubuntu 25.10

Reference host B — Windows 11 Pro for Workstations

The two hosts differ in core count, cache size and memory bandwidth. The relative ranking of the engines is identical on both; the absolute multipliers scale with the hardware. On a typical 8–16-core workstation the same comparisons remain within 10–100×.

All numbers below are means in milliseconds. Each cell is written as time (speedup×), where time is the mean for that engine and speedup× is time / dataprep_time. The dataprep column itself is the baseline, so it has no multiplier.

melt() — Ubuntu 25.10

Vary rows, 1 id + 9 value columns

Vary rows, 10 id (5 int + 5 chr)

Vary value columns, 1e3 rows, 1 id

Vary value columns, 1e3 rows, 10 id

melt() — Windows 11 Pro for Workstations

Vary rows, 1 id + 9 value columns

Vary rows, 10 id (5 int + 5 chr)

Vary value columns, 1e3 rows, 1 id

Vary value columns, 1e3 rows, 10 id

dcast() — Ubuntu 25.10

Vary n_long, 1 id, 10 levels

Vary levels, 1 id, 1e6 rows

Vary n_long, 1 id, 100 levels

Vary n_id, 1e6 rows, 10 levels

dcast() — Windows 11 Pro for Workstations

Vary n_long, 1 id, 10 levels

Vary levels, 1 id, 1e6 rows

Vary n_long, 1 id, 100 levels

Vary n_id, 1e6 rows, 10 levels

Summary of speedups

Speedup is defined as competitor mean / dataprep mean. Each table summarises every benchmark cell on that host, across all seven competitors (reshape2, data.table, tidyr, pandas, polars, dask, duckdb). Cell labels are written as n_rows × n_cols × n_id × n_val for melt() and n_long × n_id × n_levels for dcast().

Ubuntu 25.10

Windows 11 Pro for Workstations

Combined across both hosts, the smallest speedups remain at 0.6× (melt() at 1e7 rows on Ubuntu), while the largest reach 1628.9× for melt() and 799.8× for dcast(). The mean speedup is above 46× for melt() and above 76× for dcast() on both hosts. For melt() on the largest cells (1e8 rows, 1 id + 9 val, 8 GB of input), dataprep is the only engine that completes within 2.5 s, specifically < 0.3 s on Ubuntu and < 1.0 s on Windows.

Complete tables — including mean, median, and the full per-competitor gradient — are in vignette("dataprep-performance").

Cross-engine consistency

melt() and dcast() produce output identical to reshape2, data.table, tidyr, pandas, polars, dask, and duckdb on every tested shape, within tol = 1e-12:

Reproducible scripts ship under inst/ and are disabled by default so that R CMD check does not run them. A single script, benchmark_melt_dcast.R, runs both the per-tool benchmarks and the 8-engine consistency checks:

Sys.setenv(DATAPREP_RUN_BENCHMARK = "1")
source(system.file("benchmark_melt_dcast.R", package = "dataprep"))

When not to preprocess

The pipeline above assumes that the input is high-resolution instrument data with intermittent gaps and occasional outliers. Three cases where you should not run the full pipeline:

1. Already-aggregated data. data1 in this package is the result of aggregating the 61 size bins of data into three modes. It has no long gaps and no obvious outliers, so varidele, obsedele, condextr, and shorvalu have nothing to do.
2. Models that tolerate missing values. Gradient boosting, random forests, and XGBoost handle NA natively.
3. Gaps shorter than the physical mixing time. When the aerosol is well-mixed, a few missing points can be interpolated with negligible error.

See vignette("dataprep-philosophy") for the full reasoning.

Function overview

Cleaning

• varidele() — remove variables by missing fraction
• obsedele() — remove observations by consecutive missing runs
• condextr() — point-by-point weighted conditional extremum
• percoutl() — traditional percentile removal
• detect_outliers() — IQR / MAD / percentile masks
• winsorize() — cap extreme values
• phys_filter() — physical range filter
• filter_high_cor() / filter_low_var() — variable selection
• deduplicate() — exact / fuzzy duplicate removal
• validate_data() — rule-based validation
• balance_panel() — panel balancing

Missing values and imputation

• na_diagnose() — NA run statistics
• impute_missing() — linear / LOCF / NOCB / mean / median
• shorvalu() — short-period linear interpolation

Transformation

• transform_data() — log / sqrt / Box-Cox / Yeo-Johnson, z-score / min-max / robust
• log_returns() — log returns
• bin_data() — equal-width / equal-frequency / custom binning
• encode_categorical() — label / frequency / one-hot

Time series

• create_lags() — grouped lag / lead columns
• roll_apply() — rolling statistics
• resample_time() — resample to hour / day / month
• detrend_ts() — linear detrending
• remove_diurnal_cycle() — subtract mean diurnal cycle
• decompose_ts() — additive / multiplicative decomposition
• drift_detect() — rolling drift detection
• day_night_flag() / season_flag() — time flags

Reshaping

• melt() — wide to long, SIMD + OpenMP backend
• dcast() — long to wide, block-path strided copy

Workflow and reporting

• descdata() / descplot() — descriptive statistics
• percdata() / percplot() — percentile summaries
• data_report() — compact data quality report
• dry_run() — simulate preprocessing
• prep_fit() / prep_transform() — fit / transform pipeline
• sample_data() — stratified sampling

Documentation

• Design philosophy and preprocessing methodology — why the pipeline has the shape it does.
• Cleaning pipeline — step-by-step walkthrough of varidele / obsedele / condextr / shorvalu.
• Performance and cross-engine consistency — full benchmark tables and consistency checks.
• Upgrading from 0.1.5 to 0.1.8 — behaviour changes and migration checklist.
• Leakage-free workflow — prep_fit() / prep_transform().
• Fast reshaping with melt() and dcast().
• Descriptive statistics and plots.
• Function reference.
• Changelog.

Funding

This work was supported by the National Natural Science Foundation of China (No. 12301674).

Citation

If you use dataprep in published work, please cite:

Liang, C.-S., Wu, H., Li, H.-Y., Zhang, Q., Li, Z. & He, K.-B. (2020). Efficient data preprocessing, episode classification, and source apportionment of particle number concentrations. Science of the Total Environment, 741, 140923. https://doi.org/10.1016/j.scitotenv.2020.140923

License

GPL (>= 2)