get_robotstxt_http_get

domain from which to download robots.txt file

domain

HTTP user-agent string to be used to retrieve robots.txt file
from domain

user_agent

get_robotstxt() worker function to execute HTTP request

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Peter Meissner

get_robotstxt_http_get: get_robotstxt() worker function to execute HTTP request

Description

Usage

Arguments