tokenize_cpp_vec

internal

A wrapper for Jagger, a morphological analyzer proposed in Yoshinaga (2023) <arXiv:2305.19045>. Jagger uses patterns derived from morphological dictionaries and training data sets and applies them from the beginning of the input. This simultaneous and deterministic process enables it to effectively perform tokenization, POS tagging, and lemmatization.

Shusei Eshima

RcppJagger

tokenize_cpp_vec: Tokenizer (a vector input)

Description

Usage

Arguments