spiderbar (version 0.2.5)

robxp: Parse a `robots.txt` file & create a `robxp` object

Description

This function takes in a single element character vector and parses it into a `robxp` object.

Usage

robxp(x)

Value

a classed object holding an external pointer to parsed robots.txt data

Arguments

x

either an atomic character vector containing a complete `robots.txt`` file _or_ a length >1 character vector that will concatenated into a single string _or_ a `connection` object that will be passed to [readLines()], the result of which will be concatenated into a single string and parsed and the connection will be closed.

Examples

imdb <- paste0(readLines(system.file("extdata", "imdb-robots.txt",
               package="spiderbar")), collapse="\n")
rt <- robxp(imdb)