neonOS (version 1.0.0)

removeDups: Remove duplicates from a data table based on a provided primary key; flag duplicates that can't be removed.

Description

NEON observational data may contain duplicates; this function removes exact duplicates, attempts to resolve non-exact duplicates, and flags duplicates that can't be resolved.

Usage

removeDups(data, variables, table = NA_character_)

Value

A modified data frame with resolveable duplicates removed and a flag field added and populated.

Arguments

data

A data frame containing data from a NEON observational data table [data frame]

variables

The NEON variables file containing metadata about the data table in question [data frame]

table

The name of the table. Must match one of the table names in 'variables' [character]

Author

Claire Lunch

Details

Duplicates are identified based on exact matches in the values of the primary key. For records with identical keys, these steps are followed, in order: (1) If records are identical except for NA or empty string values, the non-empty values are kept. (2) If records are identical except for uid, remarks, and/or personnel (xxxxBy) fields, unique values are concatenated within each field, and the merged version is kept. (3) For records that are identical following steps 1 and 2, one record is kept and flagged with duplicateRecordQF=1. (4) Records that can't be resolved by steps 1-3 are flagged with duplicateRecordQF=2. Note that in a set of three or more duplicates, some records may be resolveable and some may not; if two or more records are left after steps 1-3, all remaining records are flagged with duplicateRecordQF=2.

References

License: GNU AFFERO GENERAL PUBLIC LICENSE Version 3, 19 November 2007

Examples

# Resolve and flag duplicates in a test dataset of foliar lignin
lig_dup <- removeDups(data=cfc_lignin_test_dups, 
                      variables=cfc_lignin_variables,
                      table="cfc_lignin")