This creates a file system "like" API based off fs
(e.g. dir_ls, file_copy, etc.) for AWS S3 storage.
s3_cacheCache AWS S3
s3_cache_bucketCached s3 bucket
s3_clientpaws s3 client
region_nameAWS region when creating new connections
profile_nameThe name of a profile to use
multipart_thresholdThreshold to use multipart
request_payerThreshold to use multipart
pidGet the process ID of the R Session
retriesnumber of retries
new()Initialize S3FileSystem class
S3FileSystem$new(
aws_access_key_id = NULL,
aws_secret_access_key = NULL,
aws_session_token = NULL,
region_name = NULL,
profile_name = NULL,
endpoint = NULL,
disable_ssl = FALSE,
multipart_threshold = fs_bytes("2GB"),
request_payer = FALSE,
anonymous = FALSE,
...
)aws_access_key_id(character): AWS access key ID
aws_secret_access_key(character): AWS secret access key
aws_session_token(character): AWS temporary session token
region_name(character): Default region when creating new connections
profile_name(character): The name of a profile to use. If not given, then the default profile is used.
endpoint(character): The complete URL to use for the constructed client.
disable_ssl(logical): Whether or not to use SSL. By default, SSL is used.
multipart_threshold(fs_bytes): Threshold to use multipart instead of standard copy and upload methods.
request_payer(logical): Confirms that the requester knows that they will be charged for the request.
anonymous(logical): Set up anonymous credentials when connecting to AWS S3.
...Other parameters within paws client.
file_chmod()Change file permissions
S3FileSystem$file_chmod(
path,
mode = c("private", "public-read", "public-read-write", "authenticated-read",
"aws-exec-read", "bucket-owner-read", "bucket-owner-full-control")
)path(character): A character vector of path or s3 uri.
mode(character): A character of the mode
character vector of s3 uri paths
file_copy()copy files
S3FileSystem$file_copy(
path,
new_path,
max_batch = fs_bytes("100MB"),
overwrite = FALSE,
...
)path(character): path to a local directory of file or a uri.
new_path(character): path to a local directory of file or a uri.
max_batch(fs_bytes): Maximum batch size being uploaded with each multipart.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_put_object
character vector of s3 uri paths
file_create()Create file on AWS S3, if file already exists it will be left unchanged.
S3FileSystem$file_create(path, overwrite = FALSE, ...)path(character): A character vector of path or s3 uri.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_put_object
character vector of s3 uri paths
file_delete()Delete files in AWS S3
S3FileSystem$file_delete(path, ...)path(character): A character vector of paths or s3 uris.
...parameters to be passed to s3_delete_objects
character vector of s3 uri paths
file_download()Downloads AWS S3 files to local
S3FileSystem$file_download(path, new_path, overwrite = FALSE, ...)path(character): A character vector of paths or uris
new_path(character): A character vector of paths to the new locations.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_get_object
character vector of s3 uri paths
file_exists()Check if file exists in AWS S3
S3FileSystem$file_exists(path)path(character) s3 path to check
logical vector if file exists
file_info()Returns file information within AWS S3 directory
S3FileSystem$file_info(path)path(character): A character vector of paths or uris.
A data.table with metadata for each file. Columns returned are as follows.
bucket_name (character): AWS S3 bucket of file
key (character): AWS S3 path key of file
uri (character): S3 uri of file
size (numeric): file size in bytes
type (character): file type (file or directory)
etag (character): An entity tag is an opague identifier
last_modified (POSIXct): Created date of file.
delete_marker (logical): Specifies retrieved a logical marker
accept_ranges (character): Indicates that a range of bytes was specified.
expiration (character): File expiration
restore (character): If file is archived
archive_status (character): Archive status
missing_meta (integer): Number of metadata entries not returned in "x-amz-meta" headers
version_id (character): version id of file
cache_control (character): caching behaviour for the request/reply chain
content_disposition (character): presentational information of file
content_encoding (character): file content encodings
content_language (character): what language the content is in
content_type (character): file MIME type
expires (POSIXct): date and time the file is no longer cacheable
website_redirect_location (character): redirects request for file to another
server_side_encryption (character): File server side encryption
metadata (list): metadata of file
sse_customer_algorithm (character): server-side encryption with a customer-provided encryption key
sse_customer_key_md5 (character): server-side encryption with a customer-provided encryption key
ssekms_key_id (character): ID of the Amazon Web Services Key Management Service
bucket_key_enabled (logical): s3 bucket key for server-side encryption with
storage_class (character): file storage class information
request_charged (character): indicates successfully charged for request
replication_status (character): return specific header if request involves a bucket that is either a source or a destination in a replication rule https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.head_object
parts_count (integer): number of count parts the file has
object_lock_mode (character): the file lock mode
object_lock_retain_until_date (POSIXct): date and time of when object_lock_mode expires
object_lock_legal_hold_status (character): file legal holding
file_move()Move files to another location on AWS S3
S3FileSystem$file_move(
path,
new_path,
max_batch = fs_bytes("100MB"),
overwrite = FALSE,
...
)path(character): A character vector of s3 uri
new_path(character): A character vector of s3 uri.
max_batch(fs_bytes): Maximum batch size being uploaded with each multipart.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_copy_object
character vector of s3 uri paths
file_size()Return file size in bytes
S3FileSystem$file_size(path)path(character): A character vector of s3 uri
file_stream_in()Streams in AWS S3 file as a raw vector
S3FileSystem$file_stream_in(path, ...)path(character): A character vector of paths or s3 uri
...parameters to be passed to s3_get_object
list of raw vectors containing the contents of the file
file_stream_out()Streams out raw vector to AWS S3 file
S3FileSystem$file_stream_out(
obj,
path,
max_batch = fs_bytes("100MB"),
overwrite = FALSE,
...
)obj(raw|character): A raw vector, rawConnection, url to be streamed up to AWS S3.
path(character): A character vector of paths or s3 uri
max_batch(fs_bytes): Maximum batch size being uploaded with each multipart.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_put_object
character vector of s3 uri paths
file_temp()return the name which can be used as a temporary file
S3FileSystem$file_temp(pattern = "file", tmp_dir = "", ext = "")pattern(character): A character vector with the non-random portion of the name.
tmp_dir(character): The directory the file will be created in.
ext(character): A character vector of one or more paths.
character vector of s3 uri paths
file_tag_delete()Delete file tags
S3FileSystem$file_tag_delete(path)path(character): A character vector of paths or s3 uri
...parameters to be passed to s3_put_object
character vector of s3 uri paths
file_tag_info()Get file tags
S3FileSystem$file_tag_info(path)path(character): A character vector of paths or s3 uri
data.table of file version metadata
bucket_name (character): AWS S3 bucket of file
key (character): AWS S3 path key of file
uri (character): S3 uri of file
size (numeric): file size in bytes
version_id (character): version id of file
tag_key (character): name of tag
tag_value (character): tag value
file_tag_update()Update file tags
S3FileSystem$file_tag_update(path, tags, overwrite = FALSE)path(character): A character vector of paths or s3 uri
tags(list): Tags to be applied
overwrite(logical): To overwrite tagging or to modify inplace. Default will modify inplace.
character vector of s3 uri paths
file_touch()Similar to fs::file_touch this does not create the file if
it does not exist. Use s3fs$file_create() to do this if needed.
S3FileSystem$file_touch(path, ...)path(character): A character vector of paths or s3 uri
...parameters to be passed to s3_copy_object
character vector of s3 uri paths
file_upload()Uploads files to AWS S3
S3FileSystem$file_upload(
path,
new_path,
max_batch = fs_bytes("100MB"),
overwrite = FALSE,
...
)path(character): A character vector of local file paths to upload to AWS S3
new_path(character): A character vector of AWS S3 paths or uri's of the new locations.
max_batch(fs_bytes): Maximum batch size being uploaded with each multipart.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_put_object
and s3_create_multipart_upload
character vector of s3 uri paths
file_url()Generate presigned url for S3 object
S3FileSystem$file_url(path, expiration = 3600L, ...)path(character): A character vector of paths or uris
expiration(numeric): The number of seconds the presigned url is valid for. By default it expires in an hour (3600 seconds)
...parameters passed to s3_get_object
return character of urls
file_version_info()Get file versions
S3FileSystem$file_version_info(path, ...)path(character): A character vector of paths or uris
...parameters to be passed to s3_list_object_versions
return data.table with file version info, columns below:
bucket_name (character): AWS S3 bucket of file
key (character): AWS S3 path key of file
uri (character): S3 uri of file
size (numeric): file size in bytes
version_id (character): version id of file
owner (character): file owner
etag (character): An entity tag is an opague identifier
last_modified (POSIXct): Created date of file.
is_file()Test for file types
S3FileSystem$is_file(path)path(character): A character vector of paths or uris
logical vector if object is a file
is_dir()Test for file types
S3FileSystem$is_dir(path)path(character): A character vector of paths or uris
logical vector if object is a directory
is_bucket()Test for file types
S3FileSystem$is_bucket(path, ...)path(character): A character vector of paths or uris
...parameters to be passed to s3_list_objects_v2
logical vector if object is a AWS S3 bucket
is_file_empty()Test for file types
S3FileSystem$is_file_empty(path)path(character): A character vector of paths or uris
logical vector if file is empty
bucket_chmod()Change bucket permissions
S3FileSystem$bucket_chmod(
path,
mode = c("private", "public-read", "public-read-write", "authenticated-read")
)path(character): A character vector of path or s3 uri.
mode(character): A character of the mode
character vector of s3 uri paths
bucket_create()Create bucket
S3FileSystem$bucket_create(
path,
region_name = NULL,
mode = c("private", "public-read", "public-read-write", "authenticated-read"),
versioning = FALSE,
...
)path(character): A character vector of path or s3 uri.
region_name(character): aws region
mode(character): A character of the mode
versioning(logical): Whether to set the bucket to versioning or not.
...parameters to be passed to s3_create_bucket
character vector of s3 uri paths
bucket_delete()Delete bucket
S3FileSystem$bucket_delete(path)path(character): A character vector of path or s3 uri.
dir_copy()Copies the directory recursively to the new location.
S3FileSystem$dir_copy(
path,
new_path,
max_batch = fs_bytes("100MB"),
overwrite = FALSE,
...
)path(character): path to a local directory of file or a uri.
new_path(character): path to a local directory of file or a uri.
max_batch(fs_bytes): Maximum batch size being uploaded with each multipart.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_put_object
and s3_create_multipart_upload
character vector of s3 uri paths
dir_create()Create empty directory
S3FileSystem$dir_create(path, overwrite = FALSE, ...)path(character): A vector of directory or uri to be created in AWS S3
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_put_object
character vector of s3 uri paths
dir_delete()Delete contents and directory in AWS S3
S3FileSystem$dir_delete(path)path(character): A vector of paths or uris to directories to be deleted.
character vector of s3 uri paths
dir_exists()Check if path exists in AWS S3
S3FileSystem$dir_exists(path = ".")path(character) aws s3 path to be checked
character vector of s3 uri paths
dir_download()Downloads AWS S3 files to local
S3FileSystem$dir_download(path, new_path, overwrite = FALSE, ...)path(character): A character vector of paths or uris
new_path(character): A character vector of paths to the new locations.
Please ensure directories end with a /.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_get_object
character vector of s3 uri paths
dir_info()Returns file information within AWS S3 directory
S3FileSystem$dir_info(
path = ".",
type = c("any", "bucket", "directory", "file"),
glob = NULL,
regexp = NULL,
invert = FALSE,
recurse = FALSE,
refresh = FALSE,
...
)path(character):A character vector of one or more paths. Can be path or s3 uri.
type(character): File type(s) to return. Default ("any") returns all AWS S3 object types.
glob(character): A wildcard pattern (e.g. *.csv), passed onto
grep() to filter paths.
regexp(character): A regular expression (e.g. [.]csv$),
passed onto grep() to filter paths.
invert(logical): If code return files which do not match.
recurse(logical): Returns all AWS S3 objects in lower sub directories
refresh(logical): Refresh cached in s3_cache.
...parameters to be passed to s3_list_objects_v2
data.table with directory metadata
bucket_name (character): AWS S3 bucket of file
key (character): AWS S3 path key of file
uri (character): S3 uri of file
size (numeric): file size in bytes
version_id (character): version id of file
etag (character): An entity tag is an opague identifier
last_modified (POSIXct): Created date of file
dir_ls()Returns file name within AWS S3 directory
S3FileSystem$dir_ls(
path = ".",
type = c("any", "bucket", "directory", "file"),
glob = NULL,
regexp = NULL,
invert = FALSE,
recurse = FALSE,
refresh = FALSE,
...
)path(character):A character vector of one or more paths. Can be path or s3 uri.
type(character): File type(s) to return. Default ("any") returns all AWS S3 object types.
glob(character): A wildcard pattern (e.g. *.csv), passed onto
grep() to filter paths.
regexp(character): A regular expression (e.g. [.]csv$),
passed onto grep() to filter paths.
invert(logical): If code return files which do not match.
recurse(logical): Returns all AWS S3 objects in lower sub directories
refresh(logical): Refresh cached in s3_cache.
...parameters to be passed to s3_list_objects_v2
character vector of s3 uri paths
dir_ls_url()Generate presigned url to list S3 directories
S3FileSystem$dir_ls_url(path, expiration = 3600L, recurse = FALSE, ...)path(character): A character vector of paths or uris
expiration(numeric): The number of seconds the presigned url is valid for. By default it expires in an hour (3600 seconds)
recurse(logical): Returns all AWS S3 objects in lower sub directories
...parameters passed to s3_list_objects_v2
return character of urls
dir_tree()Print contents of directories in a tree-like format
S3FileSystem$dir_tree(path, recurse = TRUE, ...)path(character): path A path to print the tree from
recurse(logical): Returns all AWS S3 objects in lower sub directories
...Additional arguments passed to s3_dir_ls.
character vector of s3 uri paths
dir_upload()Uploads local directory to AWS S3
S3FileSystem$dir_upload(
path,
new_path,
max_batch = fs_bytes("100MB"),
overwrite = FALSE,
...
)path(character): A character vector of local file paths to upload to AWS S3
new_path(character): A character vector of AWS S3 paths or uri's of the new locations.
max_batch(fs_bytes): Maximum batch size being uploaded with each multipart.
overwrite(logical): Overwrite files if the exist. If this is FALSE
and the file exists an error will be thrown.
...parameters to be passed to s3_put_object
and s3_create_multipart_upload
character vector of s3 uri paths
path()Constructs a s3 uri path
S3FileSystem$path(..., ext = "")...(character): Character vectors
ext(character): An optional extension to append to the generated path
character vector of s3 uri paths
path_dir()Returns the directory portion of s3 uri
S3FileSystem$path_dir(path)path(character): A character vector of paths
character vector of s3 uri paths
path_ext()Returns the last extension for a path.
S3FileSystem$path_ext(path)path(character): A character vector of paths
character s3 uri file extension
path_ext_remove()Removes the last extension and return the rest of the s3 uri.
S3FileSystem$path_ext_remove(path)path(character): A character vector of paths
character vector of s3 uri paths
path_ext_set()Replace the extension with a new extension.
S3FileSystem$path_ext_set(path, ext)path(character): A character vector of paths
ext(character): New file extension
character vector of s3 uri paths
path_file()Returns the file name portion of the s3 uri path
S3FileSystem$path_file(path)path(character): A character vector of paths
character vector of file names
path_join()Construct an s3 uri path from path vector
S3FileSystem$path_join(parts)parts(character): A character vector of one or more paths
character vector of s3 uri paths
path_split()Split s3 uri path to core components bucket, key and version id
S3FileSystem$path_split(path)path(character): A character vector of one or more paths or s3 uri
list character vectors splitting the s3 uri path in "Bucket", "Key" and "VersionId"
clear_cache()Clear S3 Cache
S3FileSystem$clear_cache(path = NULL)path(character): s3 path to be cl
clone()The objects of this class are cloneable with this method.
S3FileSystem$clone(deep = FALSE)deepWhether to make a deep clone.