tokenize_tbl

An R wrapper for Jagger's tokenizer (a tibble input)

A wrapper for Jagger, a morphological analyzer proposed in Yoshinaga (2023) <arXiv:2305.19045>. Jagger uses patterns derived from morphological dictionaries and training data sets and applies them from the beginning of the input. This simultaneous and deterministic process enables it to effectively perform tokenization, POS tagging, and lemmatization.

Shusei Eshima

RcppJagger

An R Wrapper for Jagger

Naoki Yoshinaga

tokenize_tbl function

<dl><dt>tbl</dt>
<dd>a tibble.</dd>
<dt>column</dt>
<dd>a column name of the tibble to tokenize.</dd>
<dt>model_path</dt>
<dd>a path to the model.</dd>
<dt>keep</dt>
<dd>a vector of POS(s) to keep. Default is <code>NULL</code>.</dd></dl>

Arguments

An R wrapper for Jagger's tokenizer (a tibble input) — tokenize_tbl

<dl>

<dt>tbl</dt>
<dd>a tibble.</dd>


<dt>column</dt>
<dd>a column name of the tibble to tokenize.</dd>


<dt>model_path</dt>
<dd>a path to the model.</dd>


<dt>keep</dt>
<dd>a vector of POS(s) to keep. Default is <code>NULL</code>.</dd>

</dl>

tokenize_tbl: An R wrapper for Jagger's tokenizer (a tibble input)

Description

Usage

Value

Arguments

Examples