magicTokenizer

This function provides the number of tokens in a specified text, acting as a wrapper for an internal tokenizer function.

Efficient way to design and conduct psychological experiments for testing the performance of large language models. It simplifies the process of setting up experiments and data collection via language models’ API, facilitating a smooth workflow for researchers in the field of machine behaviour.

Xufeng Duan

MacBehaviour

Behavioural Studies of Large Language Models

Shixuan Li

Zhenguang Cai

magicTokenizer function

<dl><dt>text</dt>
<dd>A character string: the text for which the number of tokens is required.</dd></dl>

Arguments

magicTokenizer — magicTokenizer

<dl>

<dt>text</dt>
<dd>A character string: the text for which the number of tokens is required.</dd>

</dl>

magicTokenizer: magicTokenizer

Description

Usage

Value

Arguments