rt_get_comments

<p>extracting comments from robots.txt</p>

internal

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Peter Meissner

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Oliver Keys

Rich Fitz John

rt_get_comments function

extracting comments from robots.txt — rt_get_comments

extracting comments from robots.txt

rt_get_comments: extracting comments from robots.txt

Description

Usage

Arguments