spiderbar (version 0.2.5)

crawl_delays: Retrieve all agent crawl delay values in a robxp robots.txt object

Description

Retrieve all agent crawl delay values in a robxp robots.txt object

Usage

crawl_delays(obj)

Value

data frame of agents and their crawl delays

Arguments

obj

robxp object

Examples

gh <- paste0(readLines(system.file("extdata", "github-robots.txt",
             package="spiderbar")), collapse="\n")
gh_rt <- robxp(gh)
crawl_delays(gh_rt)

imdb <- paste0(readLines(system.file("extdata", "imdb-robots.txt",
               package="spiderbar")), collapse="\n")
imdb_rt <- robxp(imdb)
crawl_delays(imdb_rt)