generate_to_file

Generates <code>n</code> synthetic rows in chunks of <code>chunk_size</code>, so that data sets
much larger than memory can be produced. For appendable formats (<code>csv</code>,
<code>tsv</code>, <code>ndjson</code>, optionally compressed) all chunks go to one file. For
other formats (Parquet, Feather, HDF5, fst, RDS, ...) a directory <code>path</code>
is created holding one part file per chunk (<code>part-00001.parquet</code>, ...),
which <code>read_data()</code> (or e.g. <code>arrow::open_dataset()</code>) reads as a whole.

Generates synthetic versions of numeric, categorical, mixed-type
and time-series data. Marginal distributions are modelled with
interpolated empirical quantile functions and dependence with Gaussian or
Student-t copulas (Sklar, 1959). Discrete and categorical variables enter
the copula through the distributional transform (Ruschendorf, 2009,
<doi:10.1016/j.jspi.2009.05.030>), and their latent correlations are
estimated by maximum likelihood as polychoric (Olsson, 1979,
<doi:10.1007/BF02296207>) and polyserial (Olsson, Drasgow and Dorans,
1982, <doi:10.1007/BF02294164>) correlations. Unordered categories are
placed on their latent coordinate in an order estimated from their
association with the other variables. Missing values are
modelled by missingness indicators inside the copula, so that
missingness may depend on other variables. Supports stratified models,
a closeness parameter with calibration to a target distance or
disclosure-risk level, chunked generation of very large data sets,
optional hybrid copula and conditional CART or GAM synthesis for
one continuous response (requiring 'rpart' or 'mgcv'), training-only
automatic selection of the conditional engine, experimental nominal
conditional synthesis of one unordered categorical response (requiring
'nnet'), copula vector-autoregressive and bootstrap time-series synthesis,
augmentation of small training sets for machine learning, and random
generation from multivariate distributions and elliptical and
Archimedean copulas. Fidelity, utility and disclosure risk are assessed
with marginal distances, association matrices, propensity score
mean-squared error (Snoke et al., 2018, <doi:10.1111/rssa.12358>),
confidence-interval overlap, correct attribution probability and
distance to closest record, with 'ggplot2' visualisations. Data can be
read from and written to CSV, TSV, RDS, RData, Parquet, Feather, HDF5,
JSON, NDJSON, Excel, SPSS, Stata, SAS and fst files. The
empirical-quantile and copula approach builds on ideas of the
'synthesizer' package by Mark van der Loo.

Bogdan Oancea

SynthesizerPlus

Synthetic Data Generation with Copulas, Evaluation and
Multi-Format I/O

generate_to_file function

<dl><dt>object</dt>
<dd>A fitted <code>sp_synthesizer</code>.</dd>
<dt>n</dt>
<dd>Total number of rows.</dd>
<dt>path</dt>
<dd>Output file (appendable formats) or directory (other formats).</dd>
<dt>format</dt>
<dd>Format name; inferred from <code>path</code> for appendable formats.
Required when <code>path</code> is a directory name without extension.</dd>
<dt>chunk_size</dt>
<dd>Rows per chunk.</dd>
<dt>seed</dt>
<dd>Optional random seed (the whole output is reproducible).</dd>
<dt>progress</dt>
<dd>Print progress messages?</dd>
<dt>...</dt>
<dd>Passed to <code>generate()</code> (e.g. <code>dependence</code>).</dd></dl>

Arguments

Generate a large synthetic data set directly to disk — generate_to_file

<dl>

<dt>object</dt>
<dd>A fitted <code>sp_synthesizer</code>.</dd>


<dt>n</dt>
<dd>Total number of rows.</dd>


<dt>path</dt>
<dd>Output file (appendable formats) or directory (other formats).</dd>


<dt>format</dt>
<dd>Format name; inferred from <code>path</code> for appendable formats.
Required when <code>path</code> is a directory name without extension.</dd>


<dt>chunk_size</dt>
<dd>Rows per chunk.</dd>


<dt>seed</dt>
<dd>Optional random seed (the whole output is reproducible).</dd>


<dt>progress</dt>
<dd>Print progress messages?</dd>


<dt>...</dt>
<dd>Passed to <code>generate()</code> (e.g. <code>dependence</code>).</dd>

</dl>

generate_to_file: Generate a large synthetic data set directly to disk

Description

Usage

Value

Arguments

Examples