This is the constructor function for TextReuseTextDocument
objects.
This class is used for comparing documents.
TextReuseTextDocument(
text,
file = NULL,
meta = list(),
tokenizer = tokenize_ngrams,
...,
hash_func = hash_string,
minhash_func = NULL,
keep_tokens = FALSE,
keep_text = TRUE,
skip_short = TRUE
)is.TextReuseTextDocument(x)
has_content(x)
has_tokens(x)
has_hashes(x)
has_minhashes(x)
A character vector containing the text of the document. This
argument can be skipped if supplying file
.
The path to a text file, if text
is not provided.
A list with named elements for the metadata associated with this
document. If a document is created using the text
parameter, then
you must provide an id
field, e.g., meta = list(id =
"my_id")
. If the document is created using file
, then the ID will
be created from the file name.
A function to split the text into tokens. See
tokenizers
. If value is NULL
, then tokenizing and
hashing will be skipped.
Arguments passed on to the tokenizer
.
A function to hash the tokens. See
hash_string
.
A function to create minhash signatures of the document.
See minhash_generator
.
Should the tokens be saved in the document that is returned or discarded?
Should the text be saved in the document that is returned or discarded?
Should short documents be skipped? (See details.)
An R object to check.
An object of class TextReuseTextDocument
. This object inherits
from the virtual S3 class TextDocument
in the NLP
package. It contains the following elements:
The text of the document.
The tokens created from the text.
Hashes created from the tokens.
The minhash signature of the document.
The document metadata,
including the filename (if any) in file
.
This constructor function follows a three-step process. It reads in
the text, either from a file or from memory. It then tokenizes that text.
Then it hashes the tokens. Most of the comparison functions in this package
rely only on the hashes to make the comparison. By passing FALSE
to
keep_tokens
and keep_text
, you can avoid saving those
objects, which can result in significant memory savings for large corpora.
If skip_short = TRUE
, this function will return NULL
for very
short or empty documents. A very short document is one where there are two
few words to create at least two n-grams. For example, if five-grams are
desired, then a document must be at least six words long. If no value of
n
is provided, then the function assumes a value of n = 3
. A
warning will be printed with the document ID of a skipped document.
# NOT RUN {
file <- system.file("extdata/legal/ny1850-match.txt", package = "textreuse")
doc <- TextReuseTextDocument(file = file, meta = list(id = "ny1850"))
print(doc)
meta(doc)
head(tokens(doc))
head(hashes(doc))
# }
# NOT RUN {
content(doc)
# }
Run the code above in your browser using DataLab