get_tokens

A thin wrapper around the tiktoken-rs crate, allowing to encode text into Byte-Pair-Encoding (BPE) tokens and decode tokens back to text. This is useful to understand how Large Language Models (LLMs) perceive text.

David Zimmermann

rtiktoken

A Byte-Pair-Encoding (BPE) Tokenizer for OpenAI's Large Language
Models

David Zimmermann-Kollenda

Roger Zurawicki 

Authors of the dependent Rust crates 

get_tokens function

<dl><dt>text</dt>
<dd>a character string to encode to tokens, can be a vector</dd>
<dt>model</dt>
<dd>a model to use for tokenization, either a model name, e.g., <code>gpt-4o</code>
or a tokenizer, e.g., <code>o200k_base</code>.
See also <a href="https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/src/tokenizer.rs">available tokenizers</a>.</dd></dl>

Arguments

Converts text to tokens — get_tokens

<dl>

<dt>text</dt>
<dd>a character string to encode to tokens, can be a vector</dd>


<dt>model</dt>
<dd>a model to use for tokenization, either a model name, e.g., <code>gpt-4o</code>
or a tokenizer, e.g., <code>o200k_base</code>.
See also <a href='https://github.com/zurawiki/tiktoken-rs/blob/main/tiktoken-rs/src/tokenizer.rs'>available tokenizers</a>.</dd>

</dl>

get_tokens: Converts text to tokens

Description

Usage

Value

Arguments

See Also

Examples