optisolu

Searches over a grid of <code>interval</code> and <code>times</code> values to find
the combination that yields better outlier removal than the traditional
percentile method, based on sample deletion ratio (SDR), outlier removal
ratio (ORR), and signal-to-noise ratio (SNR).

Fast, efficient, and versatile preprocessing and reshaping of tabular
and time-series data. Most heavy routines are implemented in 'C++' via
'Rcpp', with optional 'OpenMP' parallelization and 'SIMD' acceleration
('AVX2' / 'AVX-512') on supported hardware. The 0.1.8 release rewrites
the cleaning routines in 'C++' and delivers a 1.1–1146× speedup over
0.1.5. The 'melt()' and 'dcast()' reshaping functions achieve a
0.6×–1628.9× speedup for 'melt()' and a 1.9×–799.8× speedup for
'dcast()' relative to every one of the seven major alternatives in
the R and Python ecosystems, at every tested scale (from 1,000 to
100,000,000 rows), and produce output identical to 'reshape2',
'data.table', 'tidyr', 'pandas', 'polars', 'dask', and 'duckdb'.
Core preprocessing steps include variable deletion by missing
fraction, observation deletion by consecutive missing runs,
point-by-point weighted outlier removal via conditional extremum,
traditional percentile-based outlier removal, and linear
interpolation within short time periods. The package also provides
fast reshaping, descriptive statistics, missing-value diagnosis,
multiple imputation strategies, winsorization, several outlier
detection methods (IQR, MAD, percentile), data transformation and
standardization, categorical encoding, duplicate removal, data
validation, data quality reporting, and stratified sampling.
Feature-engineering helpers cover binning, high-correlation and
low-variance filtering, and string cleaning. Time-series tools
cover detrending, diurnal-cycle removal, rolling statistics, lag
creation, resampling, simple decomposition, day/night and season
flags, log returns, drift detection, and panel balancing.
Fit/transform-style machine-learning interfaces prevent data
leakage during preprocessing. Methods are based on, and improved
from: Liang, C.-S., Wu, H., Li, H.-Y., Zhang, Q., Li, Z. & He,
K.-B. (2020) <doi:10.1016/j.scitotenv.2020.140923>. This work
was supported by the National Natural Science Foundation of China
(No. 12301674).

Chun-Sheng Liang

dataprep

Fast, Efficient, and Versatile Data Preprocessing and Reshaping
with 'C++', 'OpenMP' & 'SIMD'

Hao Wu

Hai-Yan Li

Qiang Zhang

Zhanqing Li

Ke-Bin He

optisolu function

<dl> <dt>data</dt>
<dd>A data frame containing numeric columns and optionally a
 grouping column.</dd> <dt>cols</dt>
<dd>Column indices or names of numeric variables.</dd> <dt>group</dt>
<dd>Grouping column index or name.</dd> <dt>interval</dt>
<dd>Maximum interval value to test (from 1 to <code>interval</code>).</dd> <dt>times</dt>
<dd>Maximum times value to test (from 1 to <code>times</code>).</dd> <dt>top</dt>
<dd>Top percentile threshold.</dd> <dt>top.error</dt>
<dd>Error margin for the top threshold.</dd> <dt>top.magnitude</dt>
<dd>Magnitude margin for the top threshold.</dd> <dt>bottom</dt>
<dd>Bottom percentile threshold.</dd> <dt>bottom.error</dt>
<dd>Error margin for the bottom threshold.</dd> <dt>bottom.magnitude</dt>
<dd>Magnitude margin for the bottom threshold.</dd> <dt>by</dt>
<dd>Time unit used when computing observation deletion.</dd> <dt>half</dt>
<dd>Half window size in minutes for observation deletion.</dd> <dt>date_col</dt>
<dd>Time column index or name.</dd> <dt>cores</dt>
<dd>Number of CPU cores.</dd> <dt>verbose</dt>
<dd>Logical; if <code>TRUE</code>, prints progress message.</dd></dl>

Arguments

Chun-Sheng Liang &lt;chun-shengliang@qq.com&gt;

Author

Find optimal combination of interval and times for condextr — optisolu

<dl>

 <dt>data</dt>
<dd>A data frame containing numeric columns and optionally a
 grouping column.</dd>

 <dt>cols</dt>
<dd>Column indices or names of numeric variables.</dd>

 <dt>group</dt>
<dd>Grouping column index or name.</dd>

 <dt>interval</dt>
<dd>Maximum interval value to test (from 1 to <code>interval</code>).</dd>

 <dt>times</dt>
<dd>Maximum times value to test (from 1 to <code>times</code>).</dd>

 <dt>top</dt>
<dd>Top percentile threshold.</dd>

 <dt>top.error</dt>
<dd>Error margin for the top threshold.</dd>

 <dt>top.magnitude</dt>
<dd>Magnitude margin for the top threshold.</dd>

 <dt>bottom</dt>
<dd>Bottom percentile threshold.</dd>

 <dt>bottom.error</dt>
<dd>Error margin for the bottom threshold.</dd>

 <dt>bottom.magnitude</dt>
<dd>Magnitude margin for the bottom threshold.</dd>

 <dt>by</dt>
<dd>Time unit used when computing observation deletion.</dd>

 <dt>half</dt>
<dd>Half window size in minutes for observation deletion.</dd>

 <dt>date_col</dt>
<dd>Time column index or name.</dd>

 <dt>cores</dt>
<dd>Number of CPU cores.</dd>

 <dt>verbose</dt>
<dd>Logical; if <code>TRUE</code>, prints progress message.</dd>

</dl>

optisolu: Find optimal combination of interval and times for condextr

Description

Usage

Value

Arguments

Author

Details

References

Examples