is_suspect_robotstxt

function that checks if file is valid / parsable robots.txt file

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Pedro Baltazar

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Peter Meissner

Kun Ren

Oliver Keys

Rich Fitz John

is_suspect_robotstxt function

<dl><dt>text</dt>
<dd>content of a robots.txt file provides as character vector</dd></dl>

Arguments

is_suspect_robotstxt — is_suspect_robotstxt

<dl>

<dt>text</dt>
<dd>content of a robots.txt file provides as character vector</dd>

</dl>

is_suspect_robotstxt: is_suspect_robotstxt

Description

Usage

Arguments