rt_last_http

get_robotstxt_http_get

the domain to get tobots.txt. file for

domain

the user agent to use for HTTP request header

user_agent

analog to CURL option
<a href="https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html">https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html</a>
-- and might help with robots.txt file retrieval in some cases

ssl_verifypeer

storage for http request response objects
get_robotstxt() worker function to execute HTTP request

datasets

Provides functions to download and parse 'robots.txt' files.
Ultimately the package makes it easy to check if bots
(spiders, crawler, scrapers, ...) are allowed to access specific
resources on a domain.

Peter Meissner

robotstxt

A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker

Kun Ren

Oliver Keys

Rich Fitz John

rt_last_http function

An object of class <code>environment</code> of length 1.

Format

<dl><dt>domain</dt>
<dd>the domain to get tobots.txt. file for</dd>
<dt>user_agent</dt>
<dd>the user agent to use for HTTP request header</dd>
<dt>ssl_verifypeer</dt>
<dd>analog to CURL option
<a href="https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html">https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html</a>
-- and might help with robots.txt file retrieval in some cases</dd></dl>

Arguments

storage for http request response objects — rt_last_http

<dl>

<dt>domain</dt>
<dd>the domain to get tobots.txt. file for</dd>


<dt>user_agent</dt>
<dd>the user agent to use for HTTP request header</dd>


<dt>ssl_verifypeer</dt>
<dd>analog to CURL option
<a href='https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html'>https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html</a>
-- and might help with robots.txt file retrieval in some cases</dd>

</dl>

rt_last_http: storage for http request response objects

Description

Usage

Format

Arguments