rt_get_fields

regex

invert

<p>extracting permissions from robots.txt</p>

internal

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Peter Meissner

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Oliver Keys

Rich Fitz John

rt_get_fields function

extracting permissions from robots.txt — rt_get_fields

extracting permissions from robots.txt

rt_get_fields: extracting permissions from robots.txt

Description

Usage

Arguments