
Last chance! 50% off unlimited learning
Sale ends in
Collocations are a sequence of words or terms that co-occur more often than would be expected by chance. Common collocation are adjectives + nouns, nouns followed by nouns, verbs and nouns, adverbs and adjectives, verbs and prepositional phrases or verbs and adverbs. This function extracts relevant collocations and computes the following statistics on them which are indicators of how likely two terms are collocated compared to being independent.
PMI (pointwise mutual information): log2(P(w1w2) / P(w1) P(w2))
MD (mutual dependency): log2(P(w1w2)^2 / P(w1) P(w2))
LFMD (log-frequency biased mutual dependency): MD + log2(P(w1w2))
As natural language is non random - otherwise you wouldn't understand what I'm saying, most of the combinations of terms are significant. That's why these indicators of collocation are merely used to order the collocations.
collocation(x, term, group, ngram_max = 2, n_min = 2, sep = " ")
a data.frame with one row per term where the sequence of the terms correspond to
the natural order of a text. The data frame x
should also contain
the columns provided in term
and group
a character vector with 1 column from x
which indicates the term
a character vector with 1 or several columns from x
which indicates
for example a document id or a sentence id. Collocations will be computed within this
group in order not to find collocations across sentences or documents for example.
integer indicating the size of the collocations. Defaults to 2, indicating to compute bigrams. If set to 3, will find collocations of bigrams and trigrams.
integer indicating the frequency of how many times a collocation should at least occur in the data in order to be returned. Defaults to 2.
character string with the separator which will be used to paste
together
terms which are collocated. Defaults to a space: ' '.
a data.frame with columns
ngram: the number of terms which are combined
collocation: the terms which are combined
left: the left term of the collocation
right: the right term of the collocation
n: the number of times the collocation occurred in the data
n_left: the number of times the left element of the collocation occurred in the data
n_right: the number of times the right element of the collocation occurrend in the data
pmi: the pointwise mutual information
md: mutual dependency
lfmd: log-frequency biased mutual dependency
# NOT RUN {
data(brussels_reviews_anno)
x <- subset(brussels_reviews_anno, language %in% "fr")
colloc <- collocation(x, term = "lemma", group = c("doc_id", "sentence_id"),
ngram_max = 3, n_min = 10)
head(colloc, 10)
## Example on finding collocations of nouns preceded by an adjective
library(data.table)
x <- as.data.table(x)
x[, xpos_previous := txt_previous(xpos, n = 1), by = list(doc_id, sentence_id)]
x[, xpos_next := txt_next(xpos, n = 1), by = list(doc_id, sentence_id)]
x <- subset(x, (xpos %in% c("NN") & xpos_previous %in% c("JJ")) |
(xpos %in% c("JJ") & xpos_next %in% c("NN")))
colloc <- collocation(x, term = "lemma", group = c("doc_id", "sentence_id"),
ngram_max = 2, n_min = 2)
head(colloc)
# }
Run the code above in your browser using DataLab