paths_allowed

paths for which to check bot's permission, defaults to "/"

paths

Domain for which paths should be checked. Defaults to "auto".
If set to "auto" function will try to guess the domain by parsing the paths
argument. Note however, that these are educated guesses which might utterly
fail. To be on the save side, provide appropriate domains manually.

domain

HTTP user-agent string to be used to retrieve robots.txt file
from domain

user_agent

wrapper to <code><a rd-options="" href="/link/path_allowed?package=robotstxt&version=0.4.1" data-mini-rdoc="robotstxt::path_allowed">path_allowed</a></code>

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Peter Meissner

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Oliver Keys

Rich Fitz John

paths_allowed function

wrapper to <code><a rd-options='' href='path_allowed'>path_allowed</a></code>

paths_allowed: check if a bot has permissions to access page(s)

Description

Usage

Arguments

See Also