is_valid_robotstxt

<p>content of a robots.txt file provides as character vector</p>

text

<p>function that checks if file is valid / parsable robots.txt file</p>

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Peter Meissner

robotstxt

is_valid_robotstxt: function that checks if file is valid / parsable robots.txt file

Description

Usage

Arguments