Ideas for improvement:
paginate(url, head = FALSE, limit = 10000, verbose = FALSE)API endpoint
Return the data (FALSE) or the number of records (TRUE)?
Number of items per query
Whether to produce verbose output.
- Paginated endpoints return a 'X-Total-Count' header record with the total number of records. This could be used to pre-allocate space and avoid the while() loop. First call to endpoint could have limit=0 to avoid retrieving any results. - Move the JSON processing outside of the loop. Just accumulate all of the JSON into a single document. Evaluate whether this is an improvement! - Use HEAD to find expected number of results. - Could avoid final API query by checking if the number of results returned was less than limit.