tokenize_spaces_punct

Tokenise text into a sequence of words. The function uses <code><a href="/link/strsplit?package=text.alignment&version=0.1.4" data-mini-rdoc="text.alignment::strsplit">strsplit</a></code> to split text into words
by using the [:space:] and [:punct:] character classes.

Find similarities between texts using the Smith-Waterman algorithm. The algorithm performs local sequence alignment and determines similar regions between two strings.
The Smith-Waterman algorithm is explained in the paper: "Identification of common molecular subsequences" by T.F.Smith and M.S.Waterman (1981), available at <doi:10.1016/0022-2836(81)90087-5>.
This package implements the same logic for sequences of words and letters instead of molecular sequences.

Jan Wijffels

text.alignment

Text Alignment with Smith-Waterman

Vrije Universiteit Brussel - DIGI: Brussels Platform for Digital Humanities 

Lincoln Mullen

tokenize_spaces_punct function

<dl><dt>x</dt>
<dd>a character string of length 1</dd></dl>

Arguments

Tokenise text into a sequence of words. The function uses <code><a href='https://rdrr.io/r/base/strsplit.html'>strsplit</a></code> to split text into words
by using the [:space:] and [:punct:] character classes.

Tokenise text into a sequence of words — tokenize_spaces_punct

<dl>

<dt>x</dt>
<dd>a character string of length 1</dd>

</dl>

State of Data and AI Literacy Report 2025

tokenize_spaces_punct: Tokenise text into a sequence of words

Description

Usage

Value

Arguments

See Also

Examples