num_tokens_file

The function batch-wise computes the total number of tokens in a text file. 
The function returns a numeric value indicating the total
number of tokens in the file. The function can be used on very large text files.

Utilizing the 'OpenAI' API as the back end (<https://platform.openai.com/docs/api-reference>), 'TheOpenAIR' offers 'R' wrapper functions for the 'ChatGPT' endpoint and several high-level functions that enable the integration of 'ChatGPT' capabilities in diverse data-related tasks, such as data cleansing and automated analytics script generation.

Ulrich Matter

TheOpenAIR

Integrate 'OpenAI' Large Language Models into Your 'R' Workflows

Jonathan Chassot

num_tokens_file function

<dl><dt>filename</dt>
<dd>character string indicating the name of the text file to read in</dd>
<dt>batch_size</dt>
<dd>integer indicating the number of lines to read in per batch (default is 1000)</dd>
<dt>encoding</dt>
<dd>character string indicating the encoding to use (default is "cl100k_base")</dd></dl>

Arguments

Compute total number of tokens in a text file — num_tokens_file

<dl>

<dt>filename</dt>
<dd>character string indicating the name of the text file to read in</dd>


<dt>batch_size</dt>
<dd>integer indicating the number of lines to read in per batch (default is 1000)</dd>


<dt>encoding</dt>
<dd>character string indicating the encoding to use (default is "cl100k_base")</dd>

</dl>

num_tokens_file: Compute total number of tokens in a text file

Description

Usage

Value

Arguments