Learn R Programming

cleanepi: Clean and standardize epidemiological data

cleanepi is an R package designed for cleaning, curating, and standardizing epidemiological data. It streamlines various data cleaning tasks that are typically expected when working with datasets in epidemiology.

Key functionalities of cleanepi include:

  1. Removing irregularities: It removes duplicated and empty rows and columns, as well as columns with constant values.

  2. Handling missing values: It replaces missing values with the standard NA format, ensuring consistency and ease of analysis.

  3. Ensuring data integrity: It ensures the uniqueness of uniquely identified columns, thus maintaining data integrity and preventing duplicates.

  4. Date conversion: It offers functionality to convert character columns to Date format under specific conditions, enhancing data uniformity and facilitating temporal analysis. It also offers conversion of numeric values written in letters into numbers.

  5. Standardizing entries: It can standardize column entries into specified formats, promoting consistency across the dataset.

  6. Time span calculation: It calculates the time span between two elements of type Date, providing valuable demographic insights for epidemiological analysis.

cleanepi operates on data frames or similar structures like tibbles, as well as linelist objects commonly used in epidemiological research. It returns the processed data in the same format, ensuring seamless integration into existing workflows. Additionally, it generates a comprehensive report detailing the outcomes of each cleaning task.

cleanepi is developed by the Epiverse-TRACE team at the Medical Research Council The Gambia unit at the London School of Hygiene and Tropical Medicine.

Installation

cleanepi can be installed from CRAN using

install.packages("cleanepi")

The latest development version of cleanepi can be installed from GitHub.

if (!require("pak")) install.packages("pak")
pak::pak("epiverse-trace/cleanepi")
library(cleanepi)

Quick start

The main function in cleanepi is clean_data(), which internally makes call of almost all standard data cleaning functions, such as removal of empty and duplicated rows and columns, replacement of missing values, etc. However, each function can also be called independently to perform a specific task. This mechanism is explained in details in the vignette. Below is typical example of how to use the clean_data() function.

# READING IN THE TEST DATASET
test_data <- readRDS(
  system.file("extdata", "test_df.RDS", package = "cleanepi")
)

study_id

event_name

country_code

country_name

date.of.admission

dateOfBirth

date_first_pcr_positive_test

sex

PS001P2

day 0

2

Gambia

01/12/2020

06/01/1972

Dec 01, 2020

1

PS002P2

day 0

2

Gambia

28/01/2021

02/20/1952

Jan 01, 2021

1

PS004P2-1

day 0

2

Gambia

15/02/2021

06/15/1961

Feb 11, 2021

-99

PS003P2

day 0

2

Gambia

11/02/2021

11/11/1947

Feb 01, 2021

1

P0005P2

day 0

2

Gambia

17/02/2021

09/26/2000

Feb 16, 2021

2

PS006P2

day 0

2

Gambia

17/02/2021

-99

May 02, 2021

2

PB500P2

day 0

2

Gambia

28/02/2021

11/03/1989

Feb 19, 2021

1

PS008P2

day 0

2

Gambia

22/02/2021

10/05/1976

Sep 20, 2021

2

PS010P2

day 0

2

Gambia

02/03/2021

09/23/1991

Feb 26, 2021

1

PS011P2

day 0

2

Gambia

05/03/2021

02/08/1991

Mar 03, 2021

2

# READING IN THE DATA DICTIONARY
test_dictionary <- readRDS(
  system.file("extdata", "test_dictionary.RDS", package = "cleanepi")
)

options

values

grp

orders

1

male

sex

1

2

female

sex

2

# DEFINING THE CLEANING PARAMETERS
replace_missing_values <- list(target_columns = NULL, na_strings = "-99")
remove_duplicates <- list(target_columns = NULL)
standardize_dates <- list(
  target_columns = NULL,
  error_tolerance = 0.4,
  format = NULL,
  timeframe = as.Date(c("1973-05-29", "2023-05-29")),
  orders = list(
    world_named_months = c("Ybd", "dby"),
    world_digit_months = c("dmy", "Ymd"),
    US_formats = c("Omdy", "YOmd")
  )
)
standardize_subject_ids <- list(
  target_columns = "study_id",
  prefix = "PS",
  suffix = "P2",
  range = c(1, 100),
  nchar = 7
)
remove_constants <- list(cutoff = 1)
standardize_column_names <- list(
  keep = "date.of.admission",
  rename = c(DOB = "dateOfBirth")
)
to_numeric <- list(target_columns = "sex", lang = "en")
# PERFORMING THE DATA CLEANING
cleaned_data <- clean_data(
  data = test_data,
  standardize_column_names = standardize_column_names,
  remove_constants = remove_constants,
  replace_missing_values = replace_missing_values,
  remove_duplicates = remove_duplicates,
  standardize_dates = standardize_dates,
  standardize_subject_ids = standardize_subject_ids,
  to_numeric = to_numeric,
  dictionary = test_dictionary,
  check_date_sequence = NULL
)
#> ℹ Cleaning column names
#> ℹ Replacing missing values with NA
#> ℹ Removing constant columns and empty rows
#> ℹ Removing duplicated rows
#> ℹ No duplicates were found.
#> ℹ Standardizing Date columns
#> ℹ Checking subject IDs format
#> ! Detected 3 invalid subject ids at lines: "3, 5, 7".
#> ℹ You can use the `correct_subject_ids()` function to correct them.
#> ℹ Converting the following  column into numeric: sex
#> 
#> ℹ Performing dictionary-based cleaning

study_id

date.of.admission

DOB

date_first_pcr_positive_test

sex

PS001P2

2020-12-01

06/01/1972

2020-12-01

male

PS002P2

2021-01-28

02/20/1952

2021-01-01

male

PS004P2-1

2021-02-15

06/15/1961

2021-02-11

NA

PS003P2

2021-02-11

11/11/1947

2021-02-01

male

P0005P2

2021-02-17

09/26/2000

2021-02-16

female

PS006P2

2021-02-17

NA

2021-05-02

female

PB500P2

2021-02-28

11/03/1989

2021-02-19

male

PS008P2

2021-02-22

10/05/1976

2021-09-20

female

PS010P2

2021-03-02

09/23/1991

2021-02-26

male

PS011P2

2021-03-05

02/08/1991

2021-03-03

female

# EXTRACT THE DATA CLEANING REPORT
report <- attr(cleaned_data, "report")
# DISPLAY THE DATA CLEANING REPORT
print_report(report)

Vignette

browseVignettes("cleanepi")

Lifecycle

This package is currently an experimental, as defined by the RECON software lifecycle. This means that it is functional, but interfaces and functionalities may change over time, testing and documentation may be lacking.

Contributions

Contributions are welcome via pull requests.

Code of Conduct

Please note that the cleanepi project is released with a Contributor Code of Conduct. By contributing to this project, you agree to abide by its terms.

Citing this package

citation("cleanepi")
#> 
#> To cite package 'cleanepi' in publications use:
#> 
#>   Mané K, Degoot A, Ahadzie B, Mohammed N, Bah B (2025).
#>   _cleanepi: Clean and Standardize Epidemiological Data_.
#>   doi:10.5281/zenodo.11473985
#>   <https://doi.org/10.5281/zenodo.11473985>,
#>   <https://epiverse-trace.github.io/cleanepi/>.
#> 
#> A BibTeX entry for LaTeX users is
#> 
#>   @Manual{,
#>     title = {cleanepi: Clean and Standardize Epidemiological Data},
#>     author = {Karim Mané and Abdoelnaser Degoot and Bankolé Ahadzie and Nuredin Mohammed and Bubacarr Bah},
#>     year = {2025},
#>     doi = {10.5281/zenodo.11473985},
#>     url = {https://epiverse-trace.github.io/cleanepi/},
#>   }

Copy Link

Version

Install

install.packages('cleanepi')

Monthly Downloads

287

Version

1.1.0

License

MIT + file LICENSE

Issues

22

Pull Requests

4

Stars

8

Forks

3

Maintainer

Karim Mané

Last Published

March 4th, 2025

Functions in cleanepi (1.1.0)

correct_subject_ids

Correct the wrong subject IDs based on the user-provided values.
date_check_outsiders

Convert and update date values
date_get_part3

Get part3 of date value
dictionary_make_metadata

Make data dictionary for 1 field
date_detect_complex_format

Detect complex date format
date_get_part2

Get part2 of date value
convert_numeric_to_date

Convert numeric to date
convert_to_numeric

Convert columns into numeric
date_make_format

Build the auto-detected format
date_rescue_lubridate_failures

Find the dates that lubridate couldn't
date_trim_outliers

Trim dates outside of the defined timeframe
date_i_guess_and_convert

Extract date from a character vector
find_duplicates

Identify and return duplicated rows in a data frame or linelist.
tr_

Flag out what message will be translated using the potools package
date_detect_separator

Detect the special character that is the separator in the date values
date_get_format

Infer date format from a vector or characters
date_choose_first_good

Choose the first non-missing date from a data frame of dates
print_report

Generate report from data cleaning operations
print_misspelled_values

Print the detected misspelled values
scan_in_character

Scan through a character column
standardize_column_names

Standardize column names of a data frame or line list
date_get_part1

Split a string based on a pattern and return the first element of the resulting vector.
detect_to_numeric_columns

Detect the numeric columns that appears as characters due to the presence of some character values in the column.
remove_duplicates

Remove duplicates
remove_constants

Remove constant data, including empty rows, empty columns, and columns with constant values.
detect_misspelled_options

Detect misspelled options in columns to be cleaned
date_detect_simple_format

Get format from a simple Date value
date_guess

Try and guess dates from a characters
modify_default_params

Update clean_data default argument's values with the user-provided values.
date_detect_format

Detect a date format with only 1 separator
date_guess_convert

Guess if a character vector contains Date values, and convert them to date
date_process

Process date variable
date_match_format_and_column

Check whether the number of provided formats matches the number of target columns to be standardized.
numbers_only

Detects whether a string contains only numbers or not.
get_default_params

Set and return clean_data default parameters
make_unique_column_names

Make column names unique when duplicated column names are found after the transformation
retrieve_column_names

Get column names
is_date_sequence_ordered

Check order of a sequence of date-events
scan_data

Scan through a data frame and return the proportion of missing, numeric, Date, character, logical values.
timespan

Calculate time span between dates
standardize_dates

Standardize date variables
get_target_column_names

Get the names of the columns from which duplicates will be found
perform_remove_constants

Remove constant data.
replace_missing_values

Replace missing values with NA
%>%

Pipe operator
replace_with_na

Detect and replace values with NA from a vector
clean_using_dictionary

Perform dictionary-based cleaning
cleanepi-package

cleanepi: Clean and Standardize Epidemiological Data
check_subject_ids

Check whether the subject IDs comply with the expected format. When incorrect IDs are found, the function sends a warning and the user can call the correct_subject_ids function to correct them.
check_date_sequence

Checks whether the order in a sequence of date events is chronological. order.
clean_data

Clean and standardize data
add_to_dictionary

Add an element to the data dictionary
check_subject_ids_oness

Checks the uniqueness in values of the sample IDs column
add_to_report

Add an element to the report object
date_detect_day_or_month

Detect the appropriate abbreviation for day or month value
date_check_timeframe

Check date time frame
common_na_strings

Common strings representing missing values
construct_misspelled_report

Build the report for the detected misspelled values during dictionary-based data cleaning operation
date_convert

Convert characters to dates