⚠️There's a newer version (0.7.13) of this package.Take me there.
robotstxt (version 0.1.2)
A 'robots.txt' Parser and 'Webbot'/'Spider'/'Crawler'
Permissions Checker
Description
Provides a 'Robotstxt' ('R6') class and accompanying methods to
parse and check 'robots.txt' files. Data fields are provided as
data frames and vectors. Permissions can be checked by providing
path character vectors and optional bot names.