ggmlR (version 0.6.1)

GGML_GLU_OP_REGLU: GLU Operation Types

Description

Constants for GLU (Gated Linear Unit) operation types. Used with ggml_glu() and ggml_glu_split().

Usage

GGML_GLU_OP_REGLU
GGML_GLU_OP_GEGLU
GGML_GLU_OP_SWIGLU
GGML_GLU_OP_SWIGLU_OAI
GGML_GLU_OP_GEGLU_ERF
GGML_GLU_OP_GEGLU_QUICK

Value

An integer constant representing a GLU operation type

Format

Integer constants

An object of class integer of length 1.

An object of class integer of length 1.

An object of class integer of length 1.

An object of class integer of length 1.

An object of class integer of length 1.

Details

• GGML_GLU_OP_REGLU (0): ReGLU - ReLU gating

• GGML_GLU_OP_GEGLU (1): GeGLU - GELU gating (used in GPT-NeoX, Falcon)

• GGML_GLU_OP_SWIGLU (2): SwiGLU - SiLU/Swish gating (used in LLaMA, Mistral)

• GGML_GLU_OP_SWIGLU_OAI (3): SwiGLU OpenAI variant

• GGML_GLU_OP_GEGLU_ERF (4): GeGLU with exact erf implementation

• GGML_GLU_OP_GEGLU_QUICK (5): GeGLU with fast approximation

Examples

# \donttest{
GGML_GLU_OP_REGLU       # 0 - ReLU gating
GGML_GLU_OP_GEGLU       # 1 - GELU gating
GGML_GLU_OP_SWIGLU      # 2 - SiLU/Swish gating
GGML_GLU_OP_SWIGLU_OAI  # 3 - SwiGLU OpenAI
GGML_GLU_OP_GEGLU_ERF   # 4 - GELU with erf
GGML_GLU_OP_GEGLU_QUICK # 5 - Fast GELU
# }