implyr (version 0.5.0)

copy_to: Copy a (very small) local data frame to Impala

Description

copy_to inserts the contents of a local data frame into a new Impala table. copy_to is intended to be used only with very small data frames. It uses the SQL INSERT ... VALUES() technique, which is not suitable for loading large amounts of data. By default, this function will throw an error if you attempt to copy a data frame with more than 1000 row/column positions. You can increase this limit at your own risk by setting the option implyr.copy_to_size_limit to a higher number.

This package does not provide tools for loading larger amounts of local data into Impala tables. This is because Impala can query data stored in several different filesystems and storage systems (HDFS, Apache Kudu, Apache HBase, Amazon S3, Microsoft ADLS, and Dell EMC Isilon) and Impala does not include built-in capability for loading local data into these systems.

Usage

# S3 method for src_impala
copy_to(
  dest,
  df,
  name = deparse(substitute(df)),
  overwrite = FALSE,
  types = NULL,
  temporary = TRUE,
  unique_indexes = NULL,
  indexes = NULL,
  analyze = FALSE,
  external = FALSE,
  force = FALSE,
  field_terminator = NULL,
  line_terminator = NULL,
  file_format = NULL,
  ...
)

Value

An object with class tbl_impala, tbl_sql,

tbl_lazy, tbl

Arguments

dest

an object with class with class src_impala

df

a (very small) local data frame

name

name for the new Impala table

overwrite

whether to overwrite existing table data (currently ignored)

types

a character vector giving variable types to use for the columns

temporary

must be set to FALSE

unique_indexes

not used

indexes

not used

analyze

whether to run COMPUTE STATS after adding data to the new table

external

whether the new table will be externally managed

force

whether to silently continue if the table already exists

field_terminator

the deliminter to use between fields in text file data. Defaults to the ASCII control-A (hex 01) character

line_terminator

the line terminator. Defaults to "\n"

file_format

the storage format to use. Options are "TEXTFILE" (default) and "PARQUET"

...

other arguments passed on to methods

Examples

library(nycflights13)
dim(airlines) # airlines data frame is very small
# [1] 16  2

if (FALSE) {
copy_to(impala, airlines, temporary = FALSE)}