get_robotstxt

<p>domain from which to download robots.txt file</p>

domain

<p>warn about being unable to download domain/robots.txt because of
HTTP resposne status 404. If this happens,</p>

warn

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, scrapers, ...) are allowed to access specific
resources on a domain.

Peter Meissner

get_robotstxt: downloading robots.txt file

Description

Usage

Arguments