rtiktoken (version 0.0.7)

decode_tokens: Decodes tokens back to text

Description

Decodes tokens back to text

Usage

decode_tokens(tokens, model)

Value

a character string of the decoded tokens or a vector or strings

Arguments

tokens

a vector of tokens to decode, or a list of tokens

model

a model to use for tokenization, either a model name, e.g., gpt-4o or a tokenizer, e.g., o200k_base. See also available tokenizers.

See Also

model_to_tokenizer(), get_tokens()

Examples

tokens <- get_tokens("Hello World", "gpt-4o")
tokens
decode_tokens(tokens, "gpt-4o")

tokens <- get_tokens(c("Hello World", "Alice Bob Charlie"), "gpt-4o")
tokens
decode_tokens(tokens, "gpt-4o")