Creates a terminal token represented as (a row of) a parse table.
create_tokens(tokens, texts, lag_newlines = 0, spaces = 0, pos_ids,
token_before = NA, token_after = NA, indention_ref_pos_ids = NA,
indents = 0, terminal = TRUE, child = NULL)
Character vector with tokens to create.
Character vector with texts of the token to create.
Character vector with lag_newlines corresponding to the tokens.
Character vector with spaces corresponding to the tokens.
Character vector with positional id corresponding to the tokens.
Character vector corresponding to the columns
token_before
.
Character vector corresponding to the columns
token_after
.
Character vector with indention ref ids corresponding to the tokens.
Vector with indents corresponding to the tokens.
Boolean vector indicating whether a token is a terminal or not.
The children of the tokens.
Other token creators: create_pos_ids
,
validate_new_pos_ids