lex.div: Analyze lexical diversity

Description

These methods analyze the lexical diversity/complexity of a text corpus.

Usage

lex.div(txt, ...)
"lex.div"(txt, segment = 100, factor.size = 0.72, min.tokens = 9, rand.sample = 42, window = 100, case.sens = FALSE, lemmatize = FALSE, detailed = FALSE, measure = c("TTR", "MSTTR", "MATTR", "C", "R", "CTTR", "U", "S", "K", "Maas", "HD-D", "MTLD", "MTLD-MA"), char = c("TTR", "MATTR", "C", "R", "CTTR", "U", "S", "K", "Maas", "HD-D", "MTLD", "MTLD-MA"), char.steps = 5, log.base = 10, force.lang = NULL, keep.tokens = FALSE, corp.rm.class = "nonpunct", corp.rm.tag = c(), quiet = FALSE)
"lex.div"(txt, segment = 100, factor.size = 0.72, min.tokens = 9, rand.sample = 42, window = 100, case.sens = FALSE, lemmatize = FALSE, detailed = FALSE, measure = c("TTR", "MSTTR", "MATTR", "C", "R", "CTTR", "U", "S", "K", "Maas", "HD-D", "MTLD", "MTLD-MA"), char = c("TTR", "MATTR", "C", "R", "CTTR", "U", "S", "K", "Maas", "HD-D", "MTLD", "MTLD-MA"), char.steps = 5, log.base = 10, force.lang = NULL, keep.tokens = FALSE, corp.rm.class = "nonpunct", corp.rm.tag = c(), quiet = FALSE)

Arguments

txt

An object of either class kRp.tagged-class, kRp.txt.freq-class, kRp.analysis-class or kRp.txt.trans-class, containing the tagged text to be analyzed.

...

Only used for the method generic.

segment

An integer value for MSTTR, defining how many tokens should form one segment.

factor.size

A real number between 0 and 1, defining the MTLD factor size.

min.tokens

An integer value, how many tokens a full factor must at least have to be considered for the MTLD-MA result.

rand.sample

An integer value, how many tokens should be assumed to be drawn for calculating HD-D.

window

An integer value for MATTR, defining how many tokens the moving window should include.

case.sens

Logical, whether types should be counted case sensitive.

lemmatize

Logical, whether analysis should be carried out on the lemmatized tokens rather than all running word forms.

detailed

Logical, whether full details of the analysis should be calculated. This currently affects MTLD and MTLD-MA, defining if all factors should be kept in the object. This slows down calculations considerably.

measure

A character vector defining the measures which should be calculated. Valid elements are "TTR", "MSTTR", "MATTR", "C", "R", "CTTR", "U", "S", "K", "Maas", "HD-D", "MTLD" and "MTLD-MA".

char

A character vector defining whether data for plotting characteristic curves should be calculated. Valid elements are "TTR","MATTR", "C", "R", "CTTR", "U", "S", "K", "Maas", "HD-D", "MTLD" and "MTLD-MA".

char.steps

An integer value defining the stepwidth for characteristic curves, in tokens.

log.base

A numeric value defining the base of the logarithm. See log for details.

force.lang

A character string defining the language to be assumed for the text, by force. See details.

keep.tokens

Logical. If TRUE all raw tokens and types will be preserved in the resulting object, in a slot called tt. For the types, also their frequency in the analyzed text will be listed.

corp.rm.class

A character vector with word classes which should be dropped. The default value "nonpunct" has special meaning and will cause the result of kRp.POS.tags(lang, c("punct","sentc"), list.classes=TRUE) to be used.

corp.rm.tag

A character vector with POS tags which should be dropped.

quiet

Logical. If FALSE, short status messages will be shown. TRUE will also suppress all potential warnings regarding the validation status of measures.

Value

An object of class kRp.TTR-class.

Details

lex.div calculates a variety of proposed indices for lexical diversity. In the following formulae, $N$ refers to the total number of tokens, and $V$ to the number of types:

"TTR":: The ordinary Type-Token Ratio: $$TTR = \frac{V}{N}$$ Wrapper function: TTR
"MSTTR":: For the Mean Segmental Type-Token Ratio (sometimes referred to as Split TTR) tokens are split up into segments of the given size, TTR for each segment is calculated and the mean of these values returned. Tokens at the end which do not make a full segment are ignored. The number of dropped tokens is reported.
"MATTR":: The Moving-Average Type-Token Ratio (Covington & McFall, 2010) calculates TTRs for a defined number of tokens (called the "window"), starting at the beginning of the text and moving this window over the text, until the last token is reached. The mean of these TTRs is the MATTR.
"C":: Herdan's C (Herdan, 1960, as cited in Tweedie & Baayen, 1998; sometimes referred to as LogTTR): $$C = \frac{\lg{V}}{\lg{N}}$$
"R":: Guiraud's Root TTR (Guiraud, 1954, as cited in Tweedie & Baayen, 1998): $$R = \frac{V}{\sqrt{N}}$$
"CTTR":: Carroll's Corrected TTR: $$CTTR = \frac{V}{\sqrt{2N}}$$
"U":: Dugast's Uber Index (Dugast, 1978, as cited in Tweedie & Baayen, 1998): $$U = \frac{(\lg{N})^2}{\lg{N} - \lg{V}}$$
"S":: Summer's index: $$S = \frac{\lg{\lg{V}}}{\lg{\lg{N}}}$$
"K":: Yule's K (Yule, 1944, as cited in Tweedie & Baayen, 1998) is calculated by: $$K = 10^4 \times \frac{(\sum_{X=1}^{X}{{f_X}X^2}) - N}{N^2}$$ where $N$ is the number of tokens, $X$ is a vector with the frequencies of each type, and $fX$ is the frequencies for each X.
"Maas":: Maas' indices ($a$, $\lg{V_0}$ & $\lg{}_{e}{V_0}$): $$a^2 = \frac{\lg{N} - \lg{V}}{\lg{N}^2}$$ $$\lg{V_0} = \frac{\lg{V}}{\sqrt{1 - \frac{\lg{V}}{\lg{N}}^2}}$$ Earlier versions (koRpus < 0.04-12) reported $a^2$, and not $a$. The measure was derived from a formula by M\"uller (1969, as cited in Maas, 1972). $\lg{}_{e}{V_0}$ is equivalent to $\lg{V_0}$, only with $e$ as the base for the logarithms. Also calculated are $a$, $\lg{V_0}$ (both not the same as before) and $V'$ as measures of relative vocabulary growth while the text progresses. To calculate these measures, the first half of the text and the full text will be examined (see Maas, 1972, p. 67 ff. for details).
"MTLD":: For the Measure of Textual Lexical Diversity (McCarthy & Jarvis, 2010) so called factors are counted. Each factor is a subsequent stream of tokens which ends (and is then counted as a full factor) when the TTR value falls below the given factor size. The value of remaining partial factors is estimated by the ratio of their current TTR to the factor size threshold. The MTLD is the total number of tokens divided by the number of factors. The procedure is done twice, both forward and backward for all tokens, and the mean of both calculations is the final MTLD result.
"MTLD-MA":: The Moving-Average Measure of Textual Lexical Diversity (Jarvis, no year) combines factor counting and a moving window similar to MATTR: After each full factor the the next one is calculated from one token after the last starting point. This is repeated until the end of text is reached for the first time. The average of all full factor lengths is the final MTLD-MA result. Factors below the min.tokens threshold are dropped.
"HD-D":: The HD-D value can be interpreted as the idealized version of vocd-D (see McCarthy & Jarvis, 2007). For each type, the probability is computed (using the hypergeometric distribution) of drawing it at least one time when drawing randomly a certain number of tokens from the text -- 42 by default. The sum of these probabilities make up the HD-D value. The sum of probabilities relative to the drawn sample size (ATTR) is also reported.

By default, if the text has to be tagged yet, the language definition is queried by calling get.kRp.env(lang=TRUE) internally. Or, if txt has already been tagged, by default the language definition of that tagged object is read and used. Set force.lang=get.kRp.env(lang=TRUE) or to any other valid value, if you want to forcibly overwrite this default behaviour, and only then. See kRp.POS.tags for all supported languages.

References

Covington, M.A. & McFall, J.D. (2010). Cutting the Gordian Knot: The Moving-Average Type-Token Ratio (MATTR). Journal of Quantitative Linguistics, 17(2), 94--100.

Maas, H.-D., (1972). \"Uber den Zusammenhang zwischen Wortschatzumfang und L\"ange eines Textes. Zeitschrift f\"ur Literaturwissenschaft und Linguistik, 2(8), 73--96.

McCarthy, P.M. & Jarvis, S. (2007). vocd: A theoretical and empirical evaluation. Language Testing, 24(4), 459--488.

McCarthy, P.M. & Jarvis, S. (2010). MTLD, vocd-D, and HD-D: A validation study of sophisticated approaces to lexical diversity assessment. Behaviour Research Methods, 42(2), 381--392.

Tweedie. F.J. & Baayen, R.H. (1998). How Variable May a Constant Be? Measures of Lexical Richness in Perspective. Computers and the Humanities, 32(5), 323--352.

Examples

Run this code

## Not run: 
# lex.div(tagged.text)
# ## End(Not run)

Run the code above in your browser using DataLab