localLLM (version 1.1.0)

detokenize: Convert Token IDs Back to Text

Description

Converts a sequence of integer token IDs back into human-readable text. This is the inverse operation of tokenization and is typically used to convert model output tokens into text that can be displayed to users.

Usage

detokenize(model, tokens)

Value

Character string containing the decoded text corresponding to the input tokens

Arguments

model

A model object created with model_load. Must be the same model that was used for tokenization to ensure proper decoding

tokens

Integer vector of token IDs to convert back to text. These are typically generated by tokenize or generate

See Also

tokenize, generate, model_load

Examples

if (FALSE) {
# Load model
model <- model_load("path/to/model.gguf")

# Tokenize then detokenize (round-trip)
original_text <- "Hello, how are you today?"
tokens <- tokenize(model, original_text)
recovered_text <- detokenize(model, tokens)
print(recovered_text)  # Should match original_text

# Generate and display text
ctx <- context_create(model)
generated_text <- generate(ctx, "The weather is", max_tokens = 10)

# Inspect individual tokens
single_token <- c(123)  # Some token ID
token_text <- detokenize(model, single_token)
print(paste("Token", single_token, "represents:", token_text))
}