Paginator: Paginator client

Description

A client to help you paginate

Arguments

client

an object of class HttpClient, from a call to HttpClient

(character) how to paginate. Only 'query_params' supported for now. In the future will support 'link_headers' and 'cursor'. See Details.

limit_param

(character) the name of the limit parameter. Default: limit

offset_param

(character) the name of the offset parameter. Default: offset

limit

(numeric/integer) the maximum records wanted

limit_chunk

(numeric/integer) the number by which to chunk requests, e.g., 10 would be be each request gets 10 records

progress

(logical) print a progress bar, using utils::txtProgressBar. Default: FALSE.

Value

a list, with objects of class HttpResponse(). Responses are returned in the order they are passed in.

Methods to paginate

Supported now:

query_params: the most common way, so is the default. This method involves setting how many records and what record to start at for each request. We send these query parameters for you.

Supported later:

link_headers: link headers are URLS for the next/previous/last request given in the response header from the server. This is relatively uncommon, though is recommended by JSONAPI and is implemented by a well known API (GitHub).
cursor: this works by a single string given back in each response, to be passed in the subsequent response, and so on until no more records remain. This is common in Solr

Details

Methods

get(path, query, ...): make a paginated GET request
post(path, query, body, encode, ...): make a paginated POST request
put(path, query, body, encode, ...): make a paginated PUT request
patch(path, query, body, encode, ...): make a paginated PATCH request
delete(path, query, body, encode, ...): make a paginated DELETE request
head(path, ...): make a paginated HEAD request - not sure if this makes any sense or not yet
responses(): list responses - returns: a list of HttpResponse objects, empty list before requests made
parse(encoding = "UTF-8"): parse content - returns: character vector, empty character vector before requests made
status_code(): (integer) HTTP status codes - returns: numeric vector, empty numeric vector before requests made
status(): (list) HTTP status objects - returns: a list of http_code objects, empty list before requests made
content(): raw content - returns: raw list, empty list before requests made
times(): curl request times - returns: list of named numeric vectors, empty list before requests made
url_fetch(path, query): get URLs that would be sent (i.e., before executing the request). the only things that change the URL are path and query parameters; body and any curl options don't change the URL - returns: character vector of URLs

See HttpClient() for information on parameters.

Examples

Run this code

# NOT RUN {
(cli <- HttpClient$new(url = "https://api.crossref.org"))
cc <- Paginator$new(client = cli, limit_param = "rows",
   offset_param = "offset", limit = 50, limit_chunk = 10)
cc
cc$get('works')
cc
cc$responses()
cc$status()
cc$status_code()
cc$times()
cc$content()
cc$parse()
lapply(cc$parse(), jsonlite::fromJSON)

# get full URLs for each request to be made
cc$url_fetch('works')
cc$url_fetch('works', query = list(query = "NSF"))

# progress bar
(cli <- HttpClient$new(url = "https://api.crossref.org"))
cc <- Paginator$new(client = cli, limit_param = "rows",
   offset_param = "offset", limit = 50, limit_chunk = 10, 
   progress = TRUE)
cc
cc$get('works')
# }

Run the code above in your browser using DataCamp Workspace