ggmlR (version 0.6.1)

clip_grad_norm: Clip gradients by global L2 norm

Description

Rescales all gradients in grads so that their global L2 norm does not exceed max_norm. Modifies the grads environment in-place and returns the pre-clip norm.

Usage

clip_grad_norm(params, grads, max_norm)

Value

Numeric: the global L2 norm before clipping (invisibly).

Arguments

params

Named list of ag_param tensors (same as passed to optimizer).

grads

Gradient environment returned by backward().

max_norm

Maximum allowed global L2 norm.

Details

Call this after backward() and before optimizer$step().

Examples

# \donttest{
w  <- ag_param(matrix(runif(4), 2, 2))
x  <- ag_tensor(matrix(c(1, 1), 2, 1))
with_grad_tape({
  out  <- ag_matmul(w, x)
  loss <- ag_mse_loss(out, matrix(0, 2, 1))
})
grads <- backward(loss)
clip_grad_norm(list(w = w), grads, max_norm = 1.0)
# }