rt_get_fields_worker

name or names of the fields to be returned, defaults to all
fields

type

subsetting field names via regular expressions

regex

invert

internal

Provides a 'Robotstxt' ('R6') class and accompanying methods to
    parse and check 'robots.txt' files. Data fields are provided as
    data frames and vectors. Permissions can be checked by providing
    path character vectors and optional bot names.

rt_get_fields_worker: extracting robotstxt fields

Description

Usage

Arguments