nlp_index_tokens

A lightweight toolkit for text retrieval and NLP with a consistent and
predictable API organized around four actions: fetching, reading,
processing, and searching. Functions cover the full pipeline from web
data acquisition to text processing and indexing. Multiple search
strategies are supported including regex, BM25 keyword ranking, cosine
similarity, and dictionary matching. Pipe-friendly with no heavy
dependencies and all outputs are plain data frames. Also useful as a
building block for retrieval-augmented generation pipelines and
autonomous agent workflows.

Jason Timm

textpress

A Lightweight and Versatile NLP Toolkit

nlp_index_tokens function

<dl><dt>tokens</dt>
<dd>A named list of character vectors.</dd>
<dt>k1</dt>
<dd>BM25 saturation parameter.</dd>
<dt>b</dt>
<dd>BM25 length normalization.</dd>
<dt>stem</dt>
<dd>Logical; if TRUE, stems tokens.</dd></dl>

Arguments

Create a BM25 Search Index — nlp_index_tokens

<dl>

<dt>tokens</dt>
<dd>A named list of character vectors.</dd>


<dt>k1</dt>
<dd>BM25 saturation parameter.</dd>


<dt>b</dt>
<dd>BM25 length normalization.</dd>


<dt>stem</dt>
<dd>Logical; if TRUE, stems tokens.</dd>

</dl>

nlp_index_tokens: Create a BM25 Search Index

Description

Usage

Arguments