robotstxt (version 0.5.2)

get_robotstxt_http_get: get_robotstxt() worker function to execute HTTP request

Description

get_robotstxt() worker function to execute HTTP request

Usage

get_robotstxt_http_get(domain,
  user_agent = utils::sessionInfo()$R.version$version.string,
  ssl_verifypeer = 1)

Arguments

domain

domain from which to download robots.txt file

user_agent

HTTP user-agent string to be used to retrieve robots.txt file from domain

ssl_verifypeer

analog to CURL option https://curl.haxx.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html -- and might help with robots.txt file retrieval in some cases