ggmlR (version 0.6.1)

ggml_swiglu: SwiGLU (Swish/SiLU Gated Linear Unit) (Graph)

Description

Creates a graph node for SwiGLU operation. SwiGLU uses SiLU (Swish) as the activation function on the first half. CRITICAL for LLaMA, Mistral, and many modern LLMs.

Usage

ggml_swiglu(ctx, a)

Value

Tensor with half the first dimension of input

Arguments

ctx

GGML context

a

Input tensor (first dimension must be even)

Details

Formula: output = SiLU(x) * gate

Examples

# \donttest{
ctx <- ggml_init(16 * 1024 * 1024)
a <- ggml_new_tensor_2d(ctx, GGML_TYPE_F32, 8, 3)
ggml_set_f32(a, rnorm(24))
r <- ggml_swiglu(ctx, a)
graph <- ggml_build_forward_expand(ctx, r)
ggml_graph_compute(ctx, graph)
result <- ggml_get_f32(r)  # Shape: 4x3
ggml_free(ctx)
# }