sudachir (version 0.1.0)

tokenizer: Sudachi tokenizer

Description

Sudachi tokenizer

Usage

tokenizer(x, mode, instance = NULL)

Arguments

x

Input text vectors

mode

Select split mode (A, B, C)

instance

This is optional if you already have an instance of <sudachipy.tokenizer.Tokenizer> Giving them a predefined instance will speed up their execution.

Examples

# NOT RUN {
tokenizer("Tokyo, Japan", mode = "A")
# }