Request Helper Functions

knitr::opts_chunk$set( collapse = TRUE, comment = "#>" )

This vignette explains the purpose and usage of:

  • request_develop(endpoint, params, base_url)
  • request_build(method, path, params, body, token, key, base_url)
  • request_make(x, ..., user_agent)

The target audience is someone writing an R package to wrap a Google API.


Why use gargle's request helpers?

Why would the developer of a Google-API-wrapping package care about the request helpers in gargle?

You can write less code and safer code, in return for a modest investment in studying your target API. That is done by ingesting the API's so-called Discovery Document.

Hundreds of Google APIs -- the ones addressed by the API Discovery Service -- share a great deal of behaviour. By ingesting the metadata provided by this service, you can use gargle's request helpers to exploit this shared data and logic, while also decreasing the chance that you and your users will submit ill-formed requests.

The request helpers in gargle check the combined inputs from user and developer against suitably prepared API metadata:

  • If required parameters are missing, an error is thrown.
  • If unrecognized parameters are submitted, an error is thrown.
  • Parameters are automatically placed in their correct location: URL substitution, query, or body.
  • Is there something else you care about? It is possible to do more, but it would help to have concrete requests.

Google provides API libraries for several languages, including Java, Go, Python, JavaScript, Ruby and more (but not R). All of these libraries are machine-generated from the metadata provided by the API Discovery Service. It is the official recommendation to use the Discovery Document when building client libraries. The gargle package aims to implement key parts of this strategy, in a way that is also idiomatic for R and its developers.

High-level design pattern

gargle facilitates this design for API-wrapping packages:

  • A machine-assisted low-level interface driven by the Discovery Document:
    • Your package exports thin wrapper functions around gargle's helpers to form and make HTTP requests, that inject package-specific logic and data, such as an API key and user agent. This is for power users and yourself.
  • High-level, task-oriented, user-facing functions that constitute the main interface of your package.
    • These functions convert user input into the form required by the API and pass it along to your low-level interface functions.

Later, specific examples are given, using the googledrive package.

gargle's HTTP request helpers

gargle provides support for creating and sending HTTP requests via these functions:

request_develop(endpoint, params, base_url): a.k.a. The Smart One.

  • Processes the info in params relative to detailed knowledge about the endpoint, derived from an API Discovery Document.
  • Checks for required and unrecognized parameters.
  • Peels off params destined for the body into their own part.
  • Returns request data in a form that anticipates the httr::VERB() call that is on the horizon.

request_build(method, path, params, body, token, key, base_url): a.k.a. The Dumb One.

  • Typically consumes the output of request_develop(), although that is not required. It can be called directly to enjoy a few luxuries even when making one-off API calls in the absence of an ingested Discovery Document.
  • Integrates params into a URL via substitution and the query string.
  • Sends either an API key or an OAuth token, but it provides no default values or logic for either.

request_make(x, ..., user_agent): actually makes the HTTP request.

  • Typically consumes the output of request_build(), although that is not required. However, if you have enough info to form a request_make() request, you would probably just make the httr::VERB() call yourself.
  • Consults x$method to determine which httr::VERB() to call, then calls it with the rest of x, ..., and user_agent passed as arguments.

They are usually called in the above order, though they don't have to be used that way. It is also fine to ignore this part of gargle and use it only for help with auth. They are separate parts of the package.

Discovery Documents

Google's API Discovery Service "provides a lightweight, JSON-based API that exposes machine-readable metadata about Google APIs". We recommend ingesting this metadata into an R list, stored as internal data in an API-wrapping client package. Then, HTTP requests inside high-level functions can be made concisely and safely, by referring to this metadata. The combined use of this data structure and gargle's request helpers can eliminate a lot of boilerplate data and logic that are shared across Google APIs and across endpoints within an API.

The gargle package ships with some functions and scripts to facilitate the ingest of a Discovery Document. You can find these files in the gargle installation like so:

ddi_dir <- system.file("discovery-doc-ingest", package = "gargle") list.files(ddi_dir)

Main files of interest to the developer of a client package:

  • ingest-functions.R is a collection of functions for downloading and ingesting a Discovery Document.
  • drive-example.R uses those functions to ingest metadata on the Drive v3 API and store it as an internal data object for use in googledrive.

The remaining files present an analysis of the Discovery Document for the Discovery API itself (very meta!) and write files that are useful for reference. Several are included at the end of this vignette.

Why aren't the ingest functions exported by gargle? First, we regard this as functionality that is needed at development time, not install or run time. This is something you'll do every few months, probably associated with preparing a release of a wrapper package. Second, the packages that are useful for wrangling JSON and lists are not existing dependencies of gargle, so putting these function in gargle would require some unappealing compromises.

Method (or endpoint) data

Our Discovery Document ingest process leaves you with an R list. Let's assume it's available in your package's namespace as an internal object named .endpoints. Each item represents one method of the API (Google's vocabulary) or an endpoint (term I originally used in, e.g., googledrive ... maybe I should conform to Google?).

Each endpoint has an id. These ids are also used as names for the list. Examples of some ids from the Drive and Sheets APIs:


Retrieve the metadata for one endpoint by name, e.g.:


That info can be passed along to request_develop(endpoint, params, base_url), which conducts sanity checks and combines this external knowledge with the data coming from the user and developer via params.

Design suggestion: forming requests

Here's the model used in googledrive. There is a low-level request helper, googledrive::request_generate(), that is used to form every request in the package. It is exported as part of a low-level API for expert use, but most users will never know it exists.

# googledrive:: request_generate <- function(endpoint = character(), params = list(), key = NULL, token = drive_token()) { ept <- .endpoints[[endpoint]] if (is.null(ept)) { stop_glue("\nEndpoint not recognized:\n * {endpoint}") } ## modifications specific to googledrive package params$key <- key %||% params$key %||% drive_api_key() if (!is.null(ept$parameters$supportsTeamDrives)) { params$supportsTeamDrives <- TRUE } req <- gargle::request_develop(endpoint = ept, params = params) gargle::request_build( path = req$path, method = req$method, params = req$params, body = req$body, token = token ) }

The endpoint argument specifies an endpoint by its name, a.k.a. its id.

params is where the processed user input goes.

key and token refer to an API key and OAuth2 token, respectively. Both can be populated by default, but it is possible to pass them explicitly. This is also where we enforce the rule that we always send exactly one of key and token.

googledrive::request_generate() takes care of everything specific to the Drive API and the user's input and task. Then it uses gargle::request_develop() followed by gargle::request_build() to finish preparing the request.

Design suggestion: making requests

The output of gargle::request_build() specifies an HTTP request.

gargle::request_make() can be used to actually execute it.

# gargle:: request_make <- function(x, ..., user_agent = gargle_user_agent()) { stopifnot(is.character(x$method)) method <- switch( x$method, GET = httr::GET, POST = httr::POST, PATCH = httr::PATCH, PUT = httr::PUT, DELETE = httr::DELETE, abort(glue("Not a recognized HTTP method: {bt(x$method)}")) ) method( url = x$url, body = x$body, x$token, user_agent, ... ) }

request_make() consults x$method to identify the httr::VERB() and then calls it with the remainder of x, ... and the user_agent.

In googledrive we have a thin wrapper around this that injects the googledrive user agent:

# googledrive:: request_make <- function(x, ...) { gargle::request_make(x, ..., user_agent = drive_ua()) }


derived from the Discovery Document for the Discovery Service

Properties of an endpoint

cat(readLines(fs::path(ddi_dir, "method-properties-humane.txt")), sep = "\n")

API-wide endpoint parameters (taken from Discovery API but, empirically, are shared with other APIs):

cat(readLines(fs::path(ddi_dir, "api-wide-parameters-humane.txt")), sep = "\n")

Properties of an endpoint parameters:

cat(readLines(fs::path(ddi_dir, "parameter-properties-humane.txt")), sep = "\n")