collect_content: Fetch homepage HTML and text for domains

Description

Every requested domain comes back, in order, whether or not it was reachable. Failures carry a code from [fetch_error_codes()] rather than a bare `NA`.

Usage

collect_content(
  domains = NULL,
  delay = 1,
  timeout = 10,
  max_bytes = 2 * 1024^2,
  obey_robots = TRUE,
  max_crawl_delay = 30,
  max_redirects = 5,
  user_agent = rdomains_user_agent()
)

Value

A tibble with one row per input: `domain_name`, `status`, `stage`, `error_code`, `retryable`, `http_status`, `final_url`, `fetched_at`, `content_bytes`, `title`, `description`, `lang`, `text`, `n_tokens`, `page_state`, `block_vendor`, `robots_allowed`, `source_last_published`.

Arguments

domains: Character vector of domains or URLs.
delay: Minimum seconds between requests to the same host. `Crawl-delay` overrides this upward.
timeout: Per-request timeout, seconds.
max_bytes: Cap on the response body actually read.
obey_robots: Whether to fetch and honour robots.txt. Turning this off is discouraged and is your responsibility, not the package's.
max_crawl_delay: Skip a host that asks for a longer delay than this rather than sleeping on it.
max_redirects: Maximum redirect hops to follow. Every hop is re-validated, so a redirect cannot be used to reach an address the first check refused.
user_agent: Override the identifying user-agent.

Details

The crawler identifies itself as `rdomains/<version>`, obeys `robots.txt` including `Crawl-delay`, spaces requests to the same host, caps the response body, and refuses to fetch hosts that resolve to private or link-local addresses.

Examples

Run this code

if (FALSE) {
res <- collect_content(c("example.com", "wikipedia.org"))
fetch_report(res)

# retry only what is worth retrying
again <- collect_content(res$domain_name[res$retryable])
}