tokenize

An R wrapper for Jagger's tokenizer

A wrapper for Jagger, a morphological analyzer proposed in Yoshinaga (2023) <arXiv:2305.19045>. Jagger uses patterns derived from morphological dictionaries and training data sets and applies them from the beginning of the input. This simultaneous and deterministic process enables it to effectively perform tokenization, POS tagging, and lemmatization.

Shusei Eshima

RcppJagger

An R Wrapper for Jagger

Naoki Yoshinaga

tokenize function

<dl><dt>input</dt>
<dd>an input.</dd>
<dt>model_path</dt>
<dd>a path to the model.</dd>
<dt>keep</dt>
<dd>a vector of POS(s) to keep. Default is <code>NULL</code>.</dd>
<dt>concat</dt>
<dd>logical. If TRUE, the function returns a concatenated string. Default is <code>TRUE</code>.</dd></dl>

Arguments

An R wrapper for Jagger's tokenizer — tokenize

<dl>

<dt>input</dt>
<dd>an input.</dd>


<dt>model_path</dt>
<dd>a path to the model.</dd>


<dt>keep</dt>
<dd>a vector of POS(s) to keep. Default is <code>NULL</code>.</dd>


<dt>concat</dt>
<dd>logical. If TRUE, the function returns a concatenated string. Default is <code>TRUE</code>.</dd>

</dl>

tokenize: An R wrapper for Jagger's tokenizer

Description

Usage

Value

Arguments

Examples