get_robotstxt

domain from which to download robots.txt file

domain

warn about being unable to download domain/robots.txt because of

warn

if TRUE instead of using possible cached results the function will
re-download the robotstxt file
HTTP response status 404. If this happens,

force

HTTP user-agent string to be used to retrieve robots.txt file
from domain

user_agent

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

get_robotstxt: downloading robots.txt file

Description

Usage

Arguments