posDebugRcpp

Tokenizes a character vector
and returns all possible results out of the tokenization process.
The returned data.frame contains additional attributes for debug usage.

internal

A plain 'Rcpp' wrapper for 'MeCab' that can segment Chinese,
Japanese, and Korean text into tokens. The main goal of this package
is to provide an alternative to 'tidytext' using morphological
analysis.

Akiru Kato

gibasa

An Alternative 'Rcpp' Wrapper of 'MeCab'

Shogo Ichinose

Taku Kudo

Jorge Nocedal

Nippon Telegraph and Telephone Corporation 

posDebugRcpp function

<dl><dt>text</dt>
<dd>A character vector to be tokenized.</dd>
<dt>sys_dic</dt>
<dd>Character scalar; path to the system dictionary for 'MeCab'.</dd>
<dt>user_dic</dt>
<dd>Character scalar; path to the user dictionary for 'MeCab'.</dd>
<dt>partial</dt>
<dd>Logical; If <code>TRUE</code>, activates partial parsing mode.</dd>
<dt>grain_size</dt>
<dd>Integer value larger than 1.</dd></dl>

Arguments

Tokenizer for debug use — posDebugRcpp

<dl>

<dt>text</dt>
<dd>A character vector to be tokenized.</dd>


<dt>sys_dic</dt>
<dd>Character scalar; path to the system dictionary for 'MeCab'.</dd>


<dt>user_dic</dt>
<dd>Character scalar; path to the user dictionary for 'MeCab'.</dd>


<dt>partial</dt>
<dd>Logical; If <code>TRUE</code>, activates partial parsing mode.</dd>


<dt>grain_size</dt>
<dd>Integer value larger than 1.</dd>

</dl>

State of Data and AI Literacy Report 2025

posDebugRcpp: Tokenizer for debug use

Description

Value

Arguments