A fully connected variational autoencoder built and trained with torch:
the encoder maps each case to a normal distribution over k latent
dimensions, and the decoder maps a draw from it back to the features.
setup_VariationalAutoencoder(
k = 2L,
hidden_units = NULL,
activation = "relu",
norm = NULL,
dropout = 0,
input_dropout = 0,
weight_decay = 0,
input_noise = 0,
loss = "mse",
optimizer = "adamw",
lr = 0.001,
beta1 = NULL,
beta2 = NULL,
eps = NULL,
momentum = NULL,
lr_scheduler = NULL,
batch_size = NULL,
max_epochs = 100L,
patience = 10L,
max_grad_norm = NULL,
validation_fraction = 0.1,
beta = 1,
features = NULL
)VariationalAutoencoderConfig object.
Integer [1, Inf): Number of components, the width of the latent layer.
Optional Integer [1, Inf) vector: Units in each hidden layer of the encoder, input side first.
Character {"relu", "gelu", "silu", "elu", "selu", "leaky_relu", "tanh"}: Activation applied after every hidden layer.
Optional Character {"batch_norm", "layer_norm"}: Normalization applied in every hidden layer. NULL applies none.
Numeric [0, 1): Dropout probability applied after every hidden layer.
Numeric [0, 1): Probability of zeroing each standardized input during training.
Numeric [0, Inf): L2 penalty, decoupled from the gradient under the adamw optimizer.
Numeric [0, Inf): Standard deviation of the Gaussian noise added to the standardized inputs during training.
Character {"mse", "l1", "smooth_l1"}: Reconstruction loss.
Character {"adamw", "adam", "sgd", "rmsprop"}: Optimization algorithm.
Numeric (0, Inf): Learning rate.
Optional Numeric [0, 1): Exponential decay rate of the first moment estimate. Applies to the adam and adamw optimizers.
Optional Numeric [0, 1): Exponential decay rate of the second moment estimate. Applies to the adam and adamw optimizers.
Optional Numeric (0, Inf): Term added to the denominator for numerical stability. Applies to the adam, adamw and rmsprop optimizers.
Optional Numeric [0, Inf): Momentum factor. Applies to the sgd and rmsprop optimizers.
Optional Character {"step", "cosine_annealing", "one_cycle", "reduce_on_plateau"}: Learning-rate schedule. NULL holds the learning rate fixed.
Optional Integer [1, Inf): Cases per optimization step.
Integer [1, Inf): Largest number of passes over the training cases.
Integer [1, Inf): Epochs without improvement of the held-out loss before stopping early.
Optional Numeric (0, Inf): Clip the gradient norm to this value before each step. NULL does not clip.
Numeric [0, 1): Fraction of the cases held out for early stopping.
Numeric [0, Inf): Weight of the KL divergence from the prior.
Optional Character vector: Names of at least 2 distinct
feature columns to decompose. NULL decomposes all numeric features.
EDG
Objective. For each case, the reconstruction loss summed over features
plus beta times the KL divergence of the case's latent distribution from
the standard normal, summed over the k dimensions; the mean over cases is
minimized. beta = 1 is the standard variational autoencoder (with
loss = "mse", up to constants, the negative evidence lower bound of a
unit-variance Gaussian decoder); beta > 1 is a beta-VAE, trading
reconstruction for components closer to independent; beta = 0 drops the
prior and leaves a noisy autoencoder.
Components are the means of the latent distributions, so decomp,
apply_decomposition() and reconstruct are deterministic: sampling happens only in
training. reconstruct decodes the means.
Architecture, input standardization, denoising, early stopping, and the execution config's seed, threads and device work as in setup_Autoencoder. The loss reported during training and for early stopping is the full objective, so it is not on the scale of setup_Autoencoder's, which averages the reconstruction loss over features.
vae_config <- setup_VariationalAutoencoder(k = 2L, beta = 4)
vae_config
Run the code above in your browser using DataLab